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Abstract 

This  research  develops  a  deterministic  interface  evaluation  framework  (IEF)  in 
support  of  the  principles  identified  in  the  Modular  Open  Systems  Approach  (MOSA). 
Interface  evaluation  in  weapon  system  development  requires  a  Decision  Analysis  (DA) 
method  capable  of  handling  a  continuously  growing  alternative  set  and  functioning  with 
limited  availability  of  senior  decision  makers.  Value  Focused  Thinking  (VFT)  is  selected 
as  the  best  method  for  addressing  the  parameters  of  the  framework.  Using  input  from  the 
Medium  Altitude  Unmanned  Aircraft  System  program  office,  the  fundamental  objectives 
of  the  interface  evaluation  framework  are:  Meet  Schedule  Expectations,  Meet  Acquisition 
Performance  Expectations,  and  Minimize  Acquisition  Cost.  An  initial  value  threshold  is 
established  to  guide  open  interface  decisions,  based  on  assessments  of  15  historical 
decision  scenarios.  Open  interface  recommendations  for  the  15  scenarios  are  compared 
to  previous  program  decisions,  where  the  model  supports  past  decisions  for  5  of  15 
scenarios.  A  sensitivity  analysis  is  then  conducted  to  examine  the  robustness  of  the 
framework  to  changing  weights  for  cost,  schedule,  and  performance,  and  the  threshold 
for  an  open  implementation  decision.  This  evaluation  framework  provides  a  repeatable 
method  for  key  interface  evaluation  that  reflects  the  values  of  DoD  acquisition  leadership 
and  the  Open  System  Joint  Task  Force  (OSJTF). 
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INTERFACE  EVALUATION  FOR  OPEN  SYSTEM  ARCHITECTURES 


I.  Introduction 


General  Issue 

The  Open  Systems  Joint  Task  Force  identifies  modular  system  design  and  proper 
use  of  interface  implementation  methods  as  critical  elements  to  cost  effective  system 
evolution  (Open  Systems  Joint  Task  Force  [OSJTF],  2004).  A  development  team,  within 
a  system  program  office,  must  make  decisions  about  the  method  of  interface 
implementation  that  will  be  utilized  across  their  system.  While  the  OSJTF  provides 
broad  guidance  on  key  interface  identification  and  implementation  method  selection,  it 
does  not  provide  a  specific  tool  or  evaluation  metrics  (2004).  Currently,  these  decisions 
are  left  to  the  judgment  of  subject  matter  experts  or  to  contractor  discretion  which  creates 
challenges  for  consistency  and  propriety.  This  document  discusses  the  research 
conducted  in  the  area  of  decision  analysis  for  an  evaluation  framework  that  could  be  used 
to  support  Open  System  Architecture  (OSA)  decisions. 

Background 

United  States  Air  Force  weapons  systems,  specifically  Medium  Altitude 
Unmanned  Aircraft  Systems  (UAS),  are  a  disparate  collection  of  subsystems  and 
components  integrated  to  achieve  a  military  objective.  This  integration  requires  many 
decisions,  during  both  the  initial  development  and  future  modifications,  about  the 
interface  implementation  method  (IIM)  to  be  utilized  between  systems.  Current  IIM 
decision  processes  leverage  expert  judgment  or  rely  on  the  judgment  of  the  contractor. 
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DoD  Directive  5000.01,  a  guiding  document  of  the  Defense  Acquisition  System,  states, 

“a  modular,  open-systems  approach  shall  be  employed,  where  feasible”  (Office  of  the 
Undersecretary  of  Defense  for  Acquisition  Technology  &  Logistics,  2007,  p.  9). 
Utilization  of  open  interfaces  is  one  of  five  major  principles  needed  to  implement  a 
modular,  open-systems  approach  (OSJTF,  2004).  One  could  argue  that,  by  definition, 
implementation  of  an  open  interface  is  always  feasible;  but  is  it  worth  it?  Does  the  use  of 
an  open  interface  add  value,  and  if  so  does  the  value  it  adds  outweigh  the  costs,  both 
monetary  and  temporal,  of  implementing  the  interface?  A  process  and  framework  is 
needed  to  calculate  the  value  of  an  open  interface  implementation. 

UAS  Background 

The  UAS  is  a  collection  of  systems  brought  together  for  the  purpose  of 

conducting  flight  operations  with  an  aircraft  that  does  not  have  a  pilot  onboard  (UAS 

Task  Force  Airspace  Integration  Integrated  Product  Team,  2011,  p.  A2).  A  weapon 

system  must  be  adapted,  where  necessary,  to  changes  in  technology  and  adversary  tactics. 

The  term  UAS  originated  from  the  Department  of  Defense  and  encompasses  a  wide 

variety  of  systems  performing  airborne  missions  without  a  human  in  the  aircraft.  The 

terms  Remotely  Piloted  Aircraft  (RPA),  Remotely  Piloted  Vehicle  (RPV),  Remotely 

Operated  Aircraft  (ROA),  Unmanned  Aerial  Vehicle  (UAV),  Unmanned  Aircraft  (UA), 

and  drone  are  used  synonymously  with  UAS  (Greenemeier,  201  l)(Federal  Aviation 

Administration,  2013).  Though  the  definitions  of  the  terms  are  substantially  different,  for 

the  purposes  of  this  research,  equal  treatment  of  them  is  reasonable.  UAS  use  as  sensor 

and  weapons  platforms  has  increased  in  the  years  following  the  World  Trade  Center 

terrorist  attacks  of  2001.  The  attacks  on  the  World  Trade  Center  brought  about  a  shift  in 
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combat  operations  from  regular  to  irregular  warfare  (Greenemeier,  2011).  This  shift  to 
irregular  warfare  drives  a  rapidly  evolving  threat  matrix  which  changes  the  capabilities 
required  to  combat  the  threats.  The  proper  method  of  interface  implementation  is  critical 
to  accommodating  capability  change  with  minimal  impact  to  operations,  sustainment,  and 
development  schedule. 

Interface  Implementation  Methods 

Understanding  the  theoretical  levels  of  integration  is  critical  to  making  IIM 
decisions.  The  Open  Systems  Joint  Task  Force  (OSJTF)  established  the  concept  of  a  key 
interface,  which  has  attributes  that  would  benefit  from  an  open  standard  interface.  Open 
standard  interfaces  or  industry  standards  are  widely  used  and  facilitate  system  flexibility 
or  interoperability.  The  OSJTF  identifies  a  non-key  interface  as  having  attributes  that 
would  not  benefit  from  an  open  architecture  which  implies  that  a  closed  implementation 
would  suffice.  The  OSJTF  alludes  to  gradations  between  open  standard  and  closed 
interfaces  such  as  the  “proprietary  standard”  or  “de  facto  standard”  (OSJTF,  2004).  The 
gradations  identified  by  the  OSJTF  imply  the  maturity  of  the  interface  standard. 

Proper  and  consistent  IIM  decisions  are  critical  to  ensuring  USAF  technological 

dominance  in  the  future.  Improper  interface  implementation  choice  can  lead  to  increased 

cost  and  schedule  for  system  modifications,  operations,  maintenance  and  support  and  can 

ultimately  lead  to  decreased  mission  performance.  The  study  of  IIM  decisions  and  the 

creation  of  an  interface  evaluation  framework  (IEF),  to  evaluate  the  value  of  an  open 

interface  implementation,  will  enable  clarity  of  thought  for  decision  makers.  Further,  an 

IEF  will  provide  decision  makers  with  a  method  of  determining  the  value  of  an  open 

interface  which  will  help  to  ensure  proper  and  consistent  UM  decisions  are  made 
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throughout  all  levels  of  leadership.  Resultantly,  as  highlighted  by  the  OSJTF,  judicious 
application  of  the  open  systems  approach  could  decrease  weapon  system  lifecycle  cost 
and  decrease  the  schedule  associated  with  weapon  system  modification. 

Problem  Statement 

Current  decisions  regarding  IIMs  are  made  based  on  intuition  or  are  relinquished 
to  the  contractor  and  based  on  their  preference  and  contractual  strategy  which  may  not  be 
in  the  best  interest  of  the  government.  This  lack  of  a  defined  decision  process  often  leads 
to  hindsight  determinations  that  key  interfaces  would  have  benefited  from  open  standard 
methods.  Failing  to  identify  open  standard  methods  as  the  proper  IIM  decision  can  result 
in  increased  modification  cost  and  longer  integration  timelines  throughout  the  lifetime  of 
the  weapon  system.  Conversely,  open  standard  methods  applied  to  non-key  interfaces 
can  cause  resources  to  be  consumed  in  pursuit  of  a  modular  open  systems  approach 
(MOSA)  when  not  needed.  Identifying  open  standard  methods  as  the  proper  IIM  when 
not  needed  results  in  underutilization  of  an  open  standard  interface  throughout  the  system 
lifecycle. 

USAF  weapons  systems  have  many  interfaces  and  thus  require  many  IIM 
decisions  to  be  made  for  successful  fielding.  This  research  will  focus  on  an  IEF  to 
support  UAS  IIM  decisions  because  of  sponsor  interest.  The  factors  that  make  an  open 
system  interface  valuable  vary  across  the  different  perspectives  of  an  integrated  product 
team  (IPT).  The  senior  decision  maker  (SDM)  is  responsible  for  balancing  all  of  the 
influences  of  the  IPT  when  making  decisions.  Coalescing  the  IPT  perspectives  for  a 
single  IIM  decision  is  a  challenge  for  an  SDM.  This  challenge  is  intensified  when  one 
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considers  the  many  interface  decisions  required  on  a  UAS  acquisition  program. 

Currently  no  single  framework  provides  guidance  to  SDMs  for  IIM  decisions  on  UAS.  A 
framework  of  this  type  could  utilize  deterministic  evaluation  measures  derived  from  the 
values  of  the  system  program  office  to  represent  the  value  that  decision  makers  place  on 
an  open  interface.  A  clear  understanding  of  the  technical  and  programmatic  factors,  from 
many  perspectives  of  an  IPT,  is  needed.  This  collection  of  factors  can  be  used  to  create  a 
decision  model  from  the  perspective  of  a  SDM  in  UAS  acquisition.  The  model  could  be 
used  by  all  levels  of  leadership  to  ensure  that  all  factors  are  considered  in  IIM  decisions 
and  that  decisions  are  consistent  with  the  preferences  of  the  SDM,  ultimately  leading  to  a 
balance  of  schedule,  cost,  and  performance. 

Research/Investigative  Questions 

The  research  question,  below,  indicates  the  overall  focus  of  the  research  which 
will  support  defining  a  decision  tool  for  IIMs  that  complements  the  broad  guidance 
provided  in  the  MOSA  handbook. 

Research  Question: 

-  What  is  an  evaluation  framework  for  assessing  the  value  of  an  open  interface 
implementation? 

The  research  question  will  be  addressed  through  detailed  investigation  of  the  areas 
highlighted  by  the  investigative  questions  below. 


5 


Investigative  Questions: 

-  What  attributes  are  considered  when  determining  the  value  of  an  open  interface 
implementation? 

-  What  is  the  structure  of  an  open  interface  evaluation  framework;  including  the 
value  hierarchy,  single  attribute  value  functions,  weight  factors,  and  multiattribute 
value  function? 

-  What  are  the  single-attribute  value  functions  associated  with  the  IIM  value 
hierarchy? 

-  What  value  scores  align  with  an  open  IIM  selection? 

Assumptions  and  Limitations 

The  research  described  in  this  document  is  constrained  by  several  key 
assumptions  and  research  limitations.  The  first  assumption  is  that  a  planning  horizon  will 
be  utilized  to  scope  the  framework  development.  The  planning  horizon  is  a  timeframe 
that  must  be  considered  when  formulating  all  elements  of  a  decision  model.  The 
planning  horizon  can  influence  the  decision  factor  elicitation,  value  measure  bounds, 
value  function  development,  weight  factor  determination,  and  alternative  scoring.  In 
many  cases  a  temporal  element  is  associated  with  decision  factors.  It  is  important  to 
constrain  the  planning  horizon  to  ensure  that  elicited  factors  and  value  measures  account 
for  the  same  length  of  time.  As  an  example,  if  one  were  evaluating  housing  options  and 
were  considering  both  renting  and  purchasing  a  home  factors  such  as  neighborhood, 
school  district,  and  number  of  bedrooms  may  be  valued  differently  if  the  planning 
horizon  were  one  year  as  opposed  to  ten  years.  The  second  assumption  is  that  input  from 
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a  single  UAS  program  will  be  representative  of  other  UAS  programs.  The  research  will 
leverage  a  set  of  IPT-level  contributors  that  provide  a  variety  of  perspectives  on  open 
interface  decisions.  However,  the  IPT  members  and  the  SDM  will  come  from  the  same 
UAS  program.  This  is  a  deliberate  decision  to  ensure  breadth  of  input  across  IPT 
functional  areas  within  the  time  constraints  of  the  research.  The  third  assumption  is  that 
of  preferential  independence  between  the  fundamental  objectives  of  the  value  hierarchy. 
The  concept  and  implications  of  preferential  independence  are  discussed  in  detail  in 
chapter  2.  The  final  assumption  is  that  an  interface  scenario  under  consideration  will 
meet  the  technical  performance  requirements  for  the  connected  systems.  The  implication 
of  this  assumption  is  that  interface  technical  performance  is  not  considered  in  the 
evaluation  framework.  A  limitation  of  the  research  methodology  is  the  assumption  of 
certainty  of  all  decision  factors.  It  is  assumed  that  the  individual  using  the  IEF  will  have 
a  certain  answer  to  all  decision  factors  for  the  period  of  the  planning  horizon.  This  is  an 
idealistic  assumption  that  will  limit  the  applicability  of  the  model.  The  concept  of 
including  uncertainty  in  the  framework  will  be  discussed  as  part  of  chapter  2,  but  was  not 
executed  in  this  research  effort. 

Methodology  Overview 

The  research  discussed  in  this  document  will  be  conducted  in  the  following  four 
phases  utilizing  qualitative  data  collected  to  support  the  Value  Focused  Thinking  (VFT) 
decision  analysis  method:  hierarchy  development,  value  function  development,  factor 
weighting,  and  analysis.  As  an  alternative  to  VFT,  the  Analytic  Hierarchy  Process  (AHP), 
was  considered  as  a  viable  methodology  for  IEF  development.  However,  AHP  is  best 
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applied  to  one  time  decisions  with  direct  decision  maker  interaction  (Belton,  1986).  The 
ultimate  goal  of  this  research  is  to  produce  a  model  that  can  be  leveraged  at  multiple 
organizational  levels  to  support  repetitive  IIM  decisions  consistent  with  SDM  preferences 
without  direct  engagement  by  the  SDM.  VFT  was  chosen  over  AHP  because,  while  both 
methods  could  be  utilized,  the  focus  of  this  thesis  is  on  repetitive  decisions  without  direct 
SDM  involvement. 

Each  phase  of  the  research  has  specific  qualitative  data  used  by  the  analysis 
method.  The  hierarchy  development  phase  of  the  research  will  employ  IIM  value  factor 
data  from  academic  and  doctrinal  publications  (the  gold  standard)  and  elicitation  with  six 
UAS  IPT  members  (the  silver  standard)  from  various  functional  areas  including,  program 
management,  engineering,  logistics,  finance,  contracting,  and  operations  (Parnell, 
Bresnick,  Tani,  &  Johnson,  2013).  These  factors  will  be  defined,  aggregated,  and 
organized  into  an  affinity  diagram  followed  by  a  value  hierarchy.  After  the  value 
hierarchy  is  complete,  value  preference  data  elicited  from  a  UAS  SDM  will  be  used  to 
form  single  attribute  value  functions  (SAVF)  for  each  of  the  lowest  level  hierarchical 
elements.  Upon  completion  of  the  SAVF,  development  weight  factors  elicited  from  the 
UAS  SDM  using  swing  weighting  techniques  will  be  utilized  to  establish  a  Multiattribute 
Value  Function  (MAVF).  To  verify  that  framework  is  consistent  with  the  SDM’s  values, 
consistency  checks  will  be  implemented  in  each  stage  of  the  development.  The  quality  of 
the  model  will  be  validated  with  a  subjective  assessment  of  the  hierarchy  against  the 
areas  of  completeness,  non-redundancy,  decomposability,  operability,  and  conciseness 
(Kirkwood,  1996).  Finally,  the  framework  will  be  used  to  analyze  interface 
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implementation  scenarios  from  the  MQ-l/MQ-9  program  to  make  comparisons  between 
the  theoretical  model  and  historical  precedence. 

Document  Overview 

The  remainder  of  the  document  is  subdivided  into  four  chapters.  Chapter  2 
provides  a  review  of  research  conducted  in  the  areas  of  MOSA,  AHP,  VFT,  preferential 
independence,  and  uncertainty  in  multiattribute  value  models.  Chapter  3  provides  a 
detailed  account  of  the  methodology  of  the  research,  and  Chapter  4  provides  results  and 
discussion  from  the  research  conducted  in  the  previous  chapter.  Finally,  Chapter  5 
provides  recommendations  based  on  the  results  obtained  during  this  study  and  future 
research  opportunities  in  this  area. 
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II.  Literature  Review 


Modern  weapon  systems  are  complicated  collections  of  interacting  subsystems 
and  components  brought  together  to  deliver  unique  capabilities  (Office  of  the  Deputy 
Under  Secretary  of  Defense  for  Acquisition  and  Technology,  Systems  and  Software 
Engineering,  2008).  To  develop  these  weapon  systems,  the  subsystems  and  components 
must  be  connected  in  a  way  that  optimizes  total  system  performance.  To  achieve  optimal 
performance  it  is  critical  that  development  teams  consider  all  relevant  factors  to  total 
system  performance  when  making  decisions  about  the  IIM  employed. 

The  research  contained  in  this  document  is  focused  on  developing  a  framework  to 
aid  effective  and  consistent  open  interface  decision-making.  This  chapter  discusses 
relevant  and  current  literature  addressing  two  critical  elements  supporting  this  research 
effort:  open  systems  and  decision  analysis.  First,  a  detailed  exploration  of  the  MOSA  is 
conducted.  Next,  two  primary  decision  analysis  methods,  the  AHP  and  VFT,  are 
discussed.  Then  a  detailed  explanation  of  the  execution  of  the  VFT  decision  model 
development  process  is  provided.  Finally,  Chapter  2  will  conclude  with  an  examination 
of  techniques  for  the  inclusion  of  uncertainty  in  multiattribute  decision  models. 

Modular  Open  Systems  Approach 

The  OSJTF  defines  MOSA  as  “both  a  business  and  technical  strategy  for 
developing  a  new  system  or  modernizing  an  existing  one”  (2004,  p.  2).  This  section 
describes  the  benefits  of  an  open  systems  approach  in  the  systems  engineering  process. 
Then  the  overarching  defense  department  policies  associated  with  MOSA  are  outlined. 
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Finally,  a  summary  of  the  guidance  provided  by  the  OSJTF  on  MOSA  implementation  is 
provided. 

Open  Systems  Approach  in  the  Systems  Engineering  Process 

“Today,  legacy  weapon  systems  continue  to  be  developed  with  their  own,  often 
unique  and  frequently  closed,  infrastructures,  making  upgrading  or  modifying  them  over 
their  expected  lifetimes  (20  to  40  years)  both  problematic  and  expensive”  (Hanratty, 
Lightsey,  &  Larson,  2002,  p.  1).  Additionally,  Hanratty,  et  al.  (2002)  highlight  that  the 
problem  of  expensive  weapon  system  modification  is  exacerbated  by  shrinking  budgets 
and  technology  evolution  driven  by  commercial  demands.  The  authors  assert  that,  in 
addition  to  cost  savings,  the  open  systems  approach  enables  weapon  systems  to  keep  pace 
with  technology  change  and  provides  a  tactical  advantage  from  faster  integration  of  new 
technology  (Hanratty  et  al.,  2002).  The  benefits  highlighted  by  the  OSJTF  in  Figure  1 
reinforce  the  assertions  made  by  Hanratty,  et  al.  The  open  systems  approach  is  not 
intended  to  replace  the  systems  engineering  process  but,  instead,  should  be  incorporated 
into  it  to  have  the  maximum  positive  impact.  Further,  the  use  of  open  architectures 
should  not  be  applied  to  all  elements  of  a  system.  Openness  should  be  employed  where 
its  benefits  provide  a  cost,  schedule,  and/or  tactical  advantage  (OSJTF,  2004). 

OSA/MOSA  Policy 

In  a  2004  memorandum  from  the  Office  of  the  Under  Secretary  of  Defense, 
Director  for  Defense  Systems  stated  that  “A  Modular  Open  Systems  Approach. . .  is  an 
integral  part  of  the  toolset  that  will  help  DoD  achieve  its  goal  of  providing  the  joint 
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combat  capabilities  required  for  the  21st  century”  (Lamartin,  2004,  p.  1).  In  that  same 


document  Larmartin  named  the  OSJTF  as  the  lead  for  MOSA. 

The  MOSA  was  cemented  as  part  of  DoD  acquisition  policy.  The  approach  was 
part  of  systems  engineering  direction  in  DoD  Directive  5000.01,  a  document  that 
“provides  management  principles  and  mandatory  policies  and  procedures  for  managing 
all  acquisition  programs”  (Office  of  the  Undersecretary  of  Defense  for  Acquisition 
Technology  &  Logistics,  2007,  p.  4).  MOSA  was  further  reinforced,  in  DoD  Instruction 
5000.02,  with  direction  for  program  managers  to  employ  the  approach  “to  design  for 
affordable  change,  enable  evolutionary  acquisition,  and  rapidly  field  affordable  systems 
that  are  interoperable  in  the  joint  battle  space”  (Office  of  the  Undersecretary  of  Defense 
for  Acquisition  Technology  &  Logistics,  2008,  p.  79).  The  Program  Manager’s  Guide: 
A  Modular  Open  Systems  Approach  (MOSA)  to  Acquisition  was  created  by  the  OSJTF  to 
provide  acquisition  professionals  guidance  for  implementing  MOSA  (OSJTF,  2004). 

In  2013,  DoDI  5000.02  was  revised  and  the  term  MOSA  was  removed  in  favor  of 
the  term  OSA.  The  Interim  DoD  5000.02  continued  to  instruct  that  “program  managers 
will  use  open  systems  architecture  design  principles  to  support  an  open  business  model” 
(Office  of  the  Undersecretary  of  Defense  for  Acquisition  Technology  &  Logistics,  2013, 
p.  85).  The  interim  guidance  referenced  a  new  guidebook,  the  DoD  Open  Systems 
Architecture  Contract  Guidebook  for  Program  Managers ,  which  focuses  on  the  business 
aspects  of  implementing  an  OSA.  This  new  guide  provides  a  more  detailed  account  of 
contracting  methods  for  OSA,  but  references  the  2004  OSJTF  document  for  the  technical 
aspects  and  principles  of  implementing  a  MOSA  (Department  of  Defense  Open  Systems 
Architecture  Data  Rights  Team,  2013). 
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MOSA  Guidance: 


The  overarching  goal  of  the  MOSA  is  to  enable  affordable  change  through 
modular  system  design  and  employment  of  open  standards.  The  OSJTF  indicates  that 
this  is  achieved  when  MOSA  technical  strategies  are  not  only  employed,  but  incorporated 
into  the  business  strategies  of  an  organization  (OSJTF,  2004).  Figure  1  highlights  the 
vision,  principles  and  benefits  of  the  approach. 


Vision  Principles  Benefits 


MOSA  is  an 
integral  part  of  all  y 
acquisition  strategies  to 
achieve  affordable, 
evolutionary, 
and  joint  combat 
capability  * 


Establish  Enabling  Environment 


Employ  Modular  Design 


Designate  Key  Interfaces 


Select  Open  Standards 


Certify  Conformance 


Z  Ease  of  Change 

Z  Reduced  Total  Ownership  Cost 

Z  Reduced  Cycle-Time 

Z  Enabling  Joint  Integrated 

Architectures  and  Interoperability 

Z  Risk  Mitigation 


Indicators 


Figure  1:  Modular  Open  Systems  Approach  (OSJTF,  2004,  p.  3) 


“Principle  1 :  Establish  an  Enabling  Environment”  (OSJTF,  2004,  p.  11) 

This  principle  indicates  that  the  program  manager  must  build  an  integrated 
product  development  and  support  atmosphere  that  is  capable  of  supporting  modular 
design.  Supporting  modular  design  can  have  implications  for  development,  contracting, 
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test,  and  system  support.  Additionally,  modularity  requires  special  consideration  in  a 
program’s  strategic  and  management  planning  (OSJTF,  2004). 

“Principle  2:  Employ  Modular  Design”  (OSJTF,  2004,  p.  13) 

This  principle  aims  at  dividing  the  system  into  functional  elements  that  can  be 
“developed,  maintained,  and  modified  or  upgraded”  (OSJTF,  2004,  p.  5)  independently. 

A  modular  design  requires  decomposition  of  the  high  level  system  into  lower  level 
systems  and  identification  of  interfaces  between  interacting  systems. 

“Principle  3:  Designate  Key  Interfaces”  (OSJTF,  2004,  p.  14) 

The  OSJTF  defines  a  key  interface  as  an  “interface  for  which  the  preferred 
implementation  uses  an  open  standard  to  design  the  system  for  affordable  change,  ease  of 
integration,  interoperability,  commonality,  reuse  or  other  essential  considerations  such  as 
criticality  of  function”  (OSJTF,  2004,  p.  14).  The  guide  recommends  evaluating  each 
interface  based  on  the  above  qualitative  characteristics,  but  does  not  provide  any  specific 
metrics  for  key  interface  determination.  The  MOSA  guide  recommends  the  use  of  a 
work  breakdown  structure  or  a  technical  reference  model,  example  in  Figure  2,  to  help 
identification  of  potential  interfaces.  Figure  2  represents  an  example  aircraft  divided  into 
many  high  level  modules.  Each  module  can  be  further  subdivided  until  specific 
interfaces  can  be  identified.  The  organization  managing  the  aircraft  development  must 
make  a  business  decision  as  to  what  level  of  subdivision  and  subsequently  interface 
control  is  desired  (OSJTF,  2004). 
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Figure  2:  Example  Technical  Reference  Model  (OSJTF,  2004,  p.  15) 

“Principle  4:  Use  Open  Standards”  (OSJTF,  2004,  p.  16) 

The  OSJTF  defines  an  interface  standard  as  “a  standard  that  specifies  the 
physical,  functional,  and  operational  relationships  between  various  elements  (hardware 
and  software),  to  permit  interchangeability,  interconnection,  compatibility  and/or 
communications”  (2004,  p.  A2).  Additionally  open  standards  are  defined  as  “standards 
that  are  widely  used,  consensus  based,  published  and  maintained  by  recognized  industry 
standards  organizations”  (OSJTF,  2004,  p.  A3).  The  guide  indicates  that  once  key 
interfaces  are  identified  the  feasibility  and  appropriateness  of  implementing  an  open 
standard  should  be  considered.  Table  1  shows  a  list  of  factors  to  consider  when  making 
this  determination.  The  table  does  not  indicate  specific  metrics  for  the  factors  nor  an 
indication  of  each  factors  relative  importance  to  the  decision. 
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Table  1:  Open  Standard  Implementation  Decision  Factors 


Factor 

Description 

1 

“Overall  acquisition  strategy  (e.g.,  the  likelihood  that  the 
technologies/engineering  for  full  capability  still  need  to  be 
developed  and  whether  or  not  the  longer-term  requirements  are 
stable  or  addressed  as  evolving  increments.)”  (OSJTF,  2004,  p.  16) 

2 

“Need  to  take  advantage  of  competition  throughout  the  life  cycle” 
(OSJTF,  2004,  p.  16)  " 

3 

“Support  strategy  (e.g.,  the  extent  of  market  acceptance  and 
availability  of  products  that  comply  with  a  selected  standard)” 
(OSJTF,  2004,  p.  16) 

4 

“Availability,  maturity,  verification,  and  accreditation  of  standards 
for  an  interface”  (OSJTF,  2004,  p.  16) 

5 

“Need  for  minimizing  integration  risks  over  the  life  of  the  system” 
(OSJTF,  2004,  p.  16) 

6 

“The  intensity  and  magnitude  of  risks  associated  with  a  proprietary 
interface  standard”  (OSJTF,  2004,  p.  16) 

7 

“The  degree  of  dependency  on  rapidly  evolving  technology  and  the 
technology  readiness  level  for  the  components  or  items  at  both  ends 
of  an  interface”  (OSJTF,  2004,  p.  16) 

8 

“Need  for  flexibility,  modularity,  and  interface  control”  (OSJTF, 
2004,  p.  16) 

“Principle  5:  Certify  Conformance”  (OSJTF,  2004,  p.  17) 

This  principle  is  focused  on  the  validation  and  verification  of  open  standards 
implemented  on  a  weapon  system.  The  OSJTF  indicates  that  when  an  open  architecture 
is  employed  system  performance  testing  is  no  longer  sufficient.  System  testing  and 
certification  must  incorporate  testing  of  open  standard  conformance  where  applicable 
(OSJTF,  2004). 
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Decision  Analysis 

Establishing  whether  an  open  interface  implementation  is  “worth  it”  is  a  complex 

decision.  Assuming  either  an  open  or  closed  interface  implementation  would  perform 

equally  well  against  technical  requirements,  many  remaining  factors  must  be  considered, 

such  as  the  cost,  schedule  urgency,  and  amount  of  change.  These  factors  and  many 

others  impact  whether  implementing  open  is  “worth  it”  or  not.  In  an  explanation  of  why 

decision  analysis  is  valuable  Belton  (1986)  states,  “we  are  looking  for  an  approach  which 

will  aid  the  decision  maker  in  analysis  and  synthesis  of  detailed  information  in  a  way  in 

which  is  consistent  with  her  value  judgments  about  the  relative  importance  of  her 

objectives”  (p.  2).  Before  committing  to  an  “irrevocable  allocation  of  resources,” 

decision  makers  should  ensure  they  have  an  adequate  understanding  of  the  decision  under 

consideration  (Howard  &  Abbas,  2010,  p.  12).  Decision  analysis  is  a  field  focused  on 

helping  a  decision  maker  obtain  clarity  of  thought  and  understanding  for  decisions  that 

are  too  complex  to  be  addressed  with  intuition  or  simple  logic.  The  fundamental  “goal  is 

to  structure  and  simplify  the  task  of  making  hard  decisions  as  well  and  as  easily  as  the 

nature  of  the  decision  permits”  (Von  Winterfeldt  &  Edwards,  1986,  p.  2).  The  two 

primary  contributors  that  make  decisions  complex  are  uncertainty  and  multiple 

conflicting  objectives.  Uncertainty  in  decision  making  with  a  single  objective  is  a  well 

understood  concept  in  which  the  decision  maker  is  faced  with  multiple  choices  with 

unknown  future  states.  The  choice  of  future  states  will  result  in  a  gain  or  loss  of  a  single 

utility  measure  such  as  money  or  time.  The  best  decision  is  associated  with  the  highest 

expected  utility.  Multiple  conflicting  objectives  in  decision  making  is  the  concept  of 

competing  values,  objectives  and  goals  (Keeney  &  Raiffa,  1993)  (Von  Winterfeldt  & 
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Edwards,  1986).  This  research  focuses  on  the  evaluation  of  interface  scenarios  in  the 
acquisition  environment  where  new  alternatives  are  analyzed  with  consideration  of  a 
common  set  of  value  measures  or  objectives.  Belton  (1986)  indicates  that  AHP  and  VFT, 
a  version  of  Multi- Attribute  Value  (MAV)  theory  popularized  by  Keeney,  are  the  best 
approaches  for  decisions  of  this  nature  after  considering  many  other  options  such  as 
Paretian  cost  benefit  analysis,  Social  cost  benefit  analysis  and  Multi-Objective  Decision 
Modeling  techniques  applied  to  continuous  decision  problems.  Bard  strengthens  Belton’s 
assertion  by  stating  that  AHP  and  VFT  offer  “an  integrated  framework  in  which  the 
decision  maker  can  conduct  tradeoffs  among  incommensurate  criteria  without  having  to 
rely  on  a  single  measure  of  performance”  (1992,  p.  1 1 1). 

Analytic  Hierarchy  Process 

The  AHP  is  one  of  two  widely  used  approaches  in  the  field  of  discrete  multiple 

criteria  decision  making.  The  AHP  begins  with  the  creation  of  a  value  hierarchy  which 

maps  the  objectives  of  the  decision  space.  Fittle  specific  guidance  is  provided  on  the 

construction  of  the  hierarchy.  The  AHP  is  focused  on  the  process  of  establishing  scores 

and  weights  for  a  value  function  (Belton,  1986).  The  process  employs  a  semantic  scale, 

measuring  connotative  meaning,  which  is  aligned  with  a  1-9  numeric  scale,  shown  in 

Table  2,  for  use  in  the  development  of  a  pairwise  comparison  matrix  (Saaty,  1980). 

Belton’s  (1986)  comparison  of  AHP  and  simple  multiattribute  value  function  indicates 

that  AHP  is  best  used  directly  by  a  decision  maker  for  a  single  decision  scenario  because 

of  the  use  of  the  semantic  scale.  The  hierarchy  developed  for  a  single  decision  could  be 

used  for  repetitive  decisions,  but  because  the  interpretation  of  each  level  of  the  semantic 

scale  is  unique  to  the  decision  maker  that  developed  the  hierarchy,  his  or  her  involvement 
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would  still  be  required.  The  AHP  employs  a  pairwise  comparison  technique  for 


alternative  scoring  and  weight  factor  determination  (Belton,  1986).  The  technique  for 

Table  2:  Analytic  Hierarchy  Process  Semantic  Scale  (Belton,  1986)(Saaty,  1980) 


1 

Equally  Preferred 

Intermediate  values 
may  be  used  as 
appropriate. 

3 

Weak  Preference 

5 

Strong  Preference 

7 

Demonstrable  Preference 

9 

Absolute  Preference 

scoring  alternatives  involves  the  decision  maker  answering  n(n-l)/2  questions  about 
strength  of  preference,  where  n  =  the  number  of  factors  being  compared  to  fill  the 
comparison  matrix  (Bard,  1992).  A  criticism  of  pairwise  comparison  with  a  semantic 
scale  is  that  it  assumes  a  ratio  scale  for  scoring.  For  example,  if  the  decision  maker 
strongly  prefers  alternative  A  to  alternative  B,  alternative  A  would  be  scored  a  5;  if  the 
opposite  were  true,  it  would  be  scored  a  1/5.  Similar  to  alternative  scoring,  pairwise 
comparison  of  decision  criteria  is  used  to  determine  weight  factors.  The  consistent 
application  of  the  same  process,  pairwise  comparison,  is  beneficial  to  an  untrained 
decision  maker.  The  weight  factors  in  AHP  are  criticized  because  their  meaning  is  not 
readily  understood.  The  weight  factors  and  alternative  scores  are  used  to  generate  a 
prioritized  list  of  alternatives.  To  trust  the  results  of  the  determinations,  the  decision 
maker  must  use  a  consistency  index  to  ensure  consistency  of  judgment  across  the  weights 
and  alternatives.  If  consistency  has  not  been  achieved,  the  decision  makers  must  examine 
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the  assessment  inconsistencies  and  reassess  their  preferences.  A  final  limitation  of  the 
AHP  is  that  it  is  not  appropriate  for  problems  that  involve  uncertainty  (Belton,  1986). 
Value  Focused  Thinking 

VFT  is  the  second  of  two  widely  used  approaches  in  the  field  of  discrete  multiple 
criteria  decision  making  that  are  discussed  in  this  chapter.  A  fundamental  tenant  of  VFT 
is  the  focus  on  values  prior  to  the  identification  of  alternatives.  This  tenet  allows  the 
decision  makers  to  maintain  an  open  mind  about  the  alternatives  which  could  be  effective 
solutions  to  problems.  Additionally,  this  allows  the  SDM  to  develop  a  decision  model 
without  having  a  complete  set  of  alternatives.  The  mathematical  underpinning  of  VFT  is 
multiattribute  utility  theory  which  leverages  the  assumption  of  mutual  preferential 
independence  to  employ  an  additive  value  function  in  deterministic  models  (Keeney, 
1992).  The  mutual  preferential  independence  assumption  indicates  that  the  value  score  of 
a  particular  attribute  has  no  effect  on  the  value  score  of  any  other  attribute  (Von 
Winterfeldt  &  Edwards,  1986).  In  many  instances  preferential  independence  does  not 
hold  for  all  factors.  When  interaction  exists  between  factors  a  value  function  with  partial 
additivity  can  be  employed  (Keeney  &  Raiffa,  1993).  This  concept  is  explored  in  more 
detail  in  the  preferential  independence  section.  The  use  of  component  value  functions 
allows  repeated  use  of  the  model  for  similar  decisions  and  allows  individuals  other  than 
the  SDM  to  utilize  the  model  (Belton,  1986).  A  detailed  account  of  the  ten  step  VFT 
model  development  process  is  provided  following  further  examination  of  preferential 
independence.  Some  common  VFT  terms  are  provided  in  Table  3,  to  aid  the  reader  in 
comprehension  of  the  remaining  VFT  related  sections. 
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Preferential  Independence 

Mutual  Preferential  Independence  (MPI)  is  a  driving  assumption  behind  the 
additive  value  function.  This  section  explains  the  concept  of  preferential  independence, 
implications  of  scaling  constants,  MPI  verification,  and  the  use  of  value  functions  with 
partial  additivity. 

Preferential  Independence  assumes  that  a  SDMs  preferences  over  a  single 
attribute  are  unaffected  by  the  levels  of  the  other  attributes  (Von  Winterfeldt  &  Edwards, 

1986).  Given  a  value  space  that  contains  four  attributes  (xp  x2,  x3,  x4)  and  that  Xx  is  the 

attribute  under  consideration,  x/s  complementary  set  is  (x2,  x3,x4)  . 

The  additive  functional  form  is  a  specialization  of  the  multilinear  functional  form 
in  which  no  interaction  terms  exist.  MPI  is  a  condition,  where  preferential  independence 
exists  between  all  combinations  of  attributes,  which  allows  for  exclusion  of  the 
interaction  terms.  The  multilinear  functional  form  for  two  factors  is  shown  in  equation 
(1).  If  the  interaction  terms,  highlighted  by  the  red  square  are  removed,  equation  (2),  the 
additive  functional  form  is  left.  The  inclusion  of  interaction  terms  adds  great 
complexity  to  the  mathematical  representation  of  the  SDMs  preferences  and  also  adds 
significant  additional  time  and  work  to  the  data  collection  process  (Keeney  &  Raiffa, 
1993). 
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Table  3:  VFT  Terminology 


Strategic 

Objective 

“...provides  common  guidance  to  all  decisions”  (Keeney,  1992,  p. 

41).  Serves  to  guide  the  fundamental  objectives. 

Fundamental 

Objective 

“. .  .an  essential  reason  for  interest  in  the  decision  situation”  (Keeney, 
1992,  p.  34). 

Value 

What  is  important  to  the  decision  maker  (Clemen,  1996,  p.  19).  “The 
values  are  the  decomposition  of  the  fundamental  objective.  They  are 
the  building  blocks  of  the  value  hierarchy”  (Jurk,  2002,  p.  27). 

Value  Structure 

“...the  entire  set  of  evaluation  considerations,  objectives,  and 
evaluation  measures  for  a  particular  decision  analysis”  (Kirkwood, 

1996,  p.  12). 

Value  Hierarchy 

“A  value  structure  with  a  hierarchical  structure”  (Kirkwood,  1996,  p. 
12). 

Global  Weight 

The  scaling  constant  applied  to  a  lowest  level  value  that  captures  the 
individual  value’s  relative  contribution  to  the  value  score  of  an 
alternative.  All  global  weight  sum  to  one  (Keeney  &  Raiffa,  1993,  pp. 
118-123) 

Value  Measure 

Measurement  of  the  “degree  of  attainment”  of  a  value  using  a  scale 
relevant  to  the  particular  value  (Kirkwood,  1996,  p.  12) . 

Score 

The  “specific  numerical  rating  for  a  particular  alternative  with  respect 
to  a  specified  evaluation  measure”  (Kirkwood,  1996,  p.  12). 

Alternative 

The  subject  of  evaluation  that  performs  to  a  specific  level  on  all 
elements  of  the  value  structure  (Keeney  &  Raiffa,  1993,  p.  66). 

Component 

Value  Function 
(CVF) 

The  function  converts  the  value  score/s  based  on  the  value  measure 
into  a  common  scale  measured  in  value  (Keeney  &  Raiffa,  1993,  p. 

119). 

Value  Function 

A  function  that  captures  the  relative  importance  for  all  CVFs  (Keeney 
&  Raiffa,  1993,  p.  80). 
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(1) 


*v(.Tj , x2 ,x3)  =  kl-  v, (a', )  +  k2  ■  v2 (x2 )  +  k3  ■  v3 (a:3 ) 

+  kl2-v1(xl)-v2(x2)+kl3-vl(x1)-  v3  (x3 ) 

+  k23-v2(x2)-v3(x3)  +  k123  •  v, (*, )  ■  v2 (*2 )  ■  v3 (*3 ) 


Where 

v,U,)=  CVF 

k;  =  Scaling  constant 

k(>  =  Scaling  constant  for  pairwise  interaction 
kijk  =  Scaling  constant  for  triplet  interaction 

*Note:  Equation  (1)  is  a  modification  of  the  multilinear  utility  function  captured  in 
Chapter  6  of  the  Keeney  and  Raiffa  (1993)  text.  This  modification  was  executed  to  show 
the  use  of  SAVFs  rather  than  Uni-Dimensional  Utility  Functions. 


(2) 


v(xvx2,x3)  =  vl(xl)  +v2(x2)  +  v3(x3) 

The  pairwise  scaling  constants  employed  in  the  multilinear  functional  form 
indicate  the  interaction  relationship  between  the  attributes.  If  ktj  equals  zero  then  it 
indicates  that  interaction  between  the  attributes  has  no  impact  on  the  value  of  an 
alternative.  A  &..  greater  than  zero  indicates  a  complementary  relationship  between  the 
attributes.  The  complementary  relationship  is  one  in  which  more  value  is  obtained  when 
high  scores  are  achieved  for  the  interacting  attributes.  A  k..  less  than  zero  indicates  a 
substitution  relationship  between  the  attributes.  This  relationship  is  one  where  a  high 
value  score  can  be  obtained  with  a  high  level  of  either  Xt  or  X ; .  The  additional  value 

gained  from  a  high  level  of  both  Xt  and  X  j  is  less  significant  (Keeney  &  Raiffa,  1993). 
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Verification  of  MPI  requires  examining  n-1  pairs  of  attributes  while 
systematically  varying  the  complementary  set  of  attributes,  where  n  is  the  total  number  of 
attributes  (Kirkwood,  1996)(Keeney  &  Raiffa,  1993).  “In  practice,  it  would  not  be 
reasonable  to  check  directly  for  all  possible  preferential  independence  conditions” 
(Keeney  &  Raiffa,  1993).  Ting  indicated  that  identifying  natural  attribute  groups  would 
facilitate  MPI  verification  (as  cited  in  Keeney  &  Raiffa,  1993,  p.  115).  The  strategic 
objective  of  a  hierarchy  may  be  broken  into  several  fundamental  objectives,  each  of 
which  is  most  likely  broken  down  further.  If  MPI  can  be  determined  between  the  groups, 
then  an  additive  value  function  can  be  employed  between  the  fundamental  objective 
groups.  This  technique  can  be  employed  from  the  top  down  in  a  value  hierarchy 
examining  the  sub-objectives  of  each  fundamental  objective.  This  systematic 
examination  expedites  the  determination  of  MPI  (Keeney  &  Raiffa,  1993). 

In  the  event  that  MPI  does  not  exist,  value  functions  with  partial  additivity  can  be 
employed  (Keeney  &  Raiffa,  1993).  A  way  to  illustrate  this  is  with  an  example. 

Consider  a  simple  value  hierarchy  with  the  strategic  objective  divided  into  two 

fundamental  objectives  (X,  Y).  Fundamental  objective  X  has  sub-objectives  X{  and  X2 . 

Fundamental  objective  Y  has  sub-objectives  Jj  and  Y2  .  This  example  demonstrates  the 

the  simplification  achived  with  the  MPI  assumption.  The  existence  the  partial 
preferential  independence  condition  enables  the  development  of  a  greatly  simplified 
value  function. 
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Without  MPI 


V(xrx2,  y„  y2)  =  v(x,)  +  v(x2)  +  v(y,)  +  v(y2)  +  (k  rL[2  -  v(x,)  •  v(x2))+ (k[lyl-  v(x,)  •  v(y,))  +  (kdv2-  v(x,)  •  v(y2)) 

+  (k*2,r  v(*2)  ■  VM + ■  v(y2))  +  (kyly2-  v(y,)  •  v(y2))  +  (kjU2yl-  v(x,)  •  v(x2)  •  v(y,)) 
+  (k^2y2- v(*i)  •  v(*2)  •  v(y2)) + (Klylyz VW ■  v(y,)  •  v(y2))  +  (kj2yly2-  v(x2)  •  v(y,)  •  v(y2)) 

+(k,™-v(^)-v^2)'v(y.)-v(y2)) 


With  MPI 

P(x1,x2,y1,y2)=v(xl)  +  v(x2)  +  v(yI)  +  v(y2) 

With  preferential  independence  between  fundamental  objectives 

V{xl,x2,yl,y2)  =  v{xvx2)  +  \{yl,y2) 

V(ji,j2,yi,y2)  =  (k,rvUi)  +  k,2-vU2)+kt-v(j1)-v(j2))  +  (kvl-v(y1)  +  kv2-v(y2)+k/v(y1)-v(y2)) 


VFT  Model  Development  Process 

VFT  model  development  follows  a  10  step  model  development  process.  A  flow 
chart  shown  in  Figure  3,  modified  from  that  created  by  Shoviak  (2001),  depicts  the 
sequenced  activities  of  VFT  model  development  (p.  63).  This  process  provides  the 
decision  maker  or  facilitator  with  a  guide  to  navigate  VFT  utilization.  Each  step  of  the 
process  is  explained  in  detail  in  the  following  sections. 


25 


Step  1 :  Problem  Identification 


The  problem  identification  step  is  intended  to  ensure  complete  understanding  of  the 
decision  under  consideration.  Establishing  the  decision  frame  facilitates  this 
understanding  and  allows  the  decision  maker  to  determine  the  factors  that  are  important 


Figure  3:  Decision  Support  Model  Development  Framework  (Shoviak,  2001) 


to  evaluating  a  decision.  The  decision  frame  is  comprised  of  the  decision  context  and  the 
fundamental  objectives.  These  elements  provide  the  guidance  upon  which  the  decision 
model  will  be  based  and  establish  decision  boundaries  such  as  time  horizon  and 
perspective  (Keeney,  1992). 
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Step  2:  Create  Value  Hierarchy 

The  value  hierarchy  is  a  structured  representation  of  salient  factors  to  a  decision 
maker  with  respect  to  a  particular  decision.  The  hierarchy  is  constructed  with  a  strategic 
objective,  overall  goal,  at  the  top  level  and  then  sub-divided  in  more  detailed  lower-level 
objectives  until  all  relevant  factors  are  depicted  (Keeney  &  Raiffa,  1993).  The  goal  of  a 
finished  hierarchy  is  to  be  complete,  non-redundant,  decomposable,  operable,  and 
concise.  To  be  complete,  the  hierarchy  must  encompass  all  germane  factors  to  a  decision 
situation.  Non-redundancy  focuses  to  ensure  that  the  same  or  similar  factors  are  not  used 
twice  in  the  same  tier  of  the  hierarchy.  A  hierarchy  is  decomposable  if  its  factors  are 
independent,  meaning  that  one  factor  does  not  influence  the  value  judgment  of  another 
factor.  The  term  operable  indicates  that  the  factors  of  the  hierarchy  are  understood  by  the 
user.  Finally,  the  concept  of  conciseness  is  to  keep  the  hierarchy  as  small  as  possible  to 
facilitate  SDM  understanding  (Kirkwood,  1996).  In  addition  to  the  five  attributes  of  a 
good  hierarchy  listed  above,  a  sixth  attribute,  input  quality  contributes  to  the  credibility 
of  an  evaluation  framework.  Three  approaches  to  obtaining  the  information  to  build  the 
value  hierarchy  have  been  described.  The  silver  standard  approach,  regarded  as  the  least 
desirable  of  the  three,  requires  elicitation  of  decision  factors  from  decision  maker 
representatives.  This  approach  is  typically  utilized  if  the  decision  maker  is  time 
constrained  or  unavailable  to  provide  direct  input.  The  gold  standard  approach  involves 
obtaining  the  information  from  doctrinal  documentation  such  as  vision  statements, 
operating  instructions,  or  strategic  plans.  Finally,  in  the  platinum  standard  approach 
hierarchy  inputs  are  elicited  directly  from  the  decision  maker  and  can  be  regarded  as  the 

most  accurate  representation  of  their  preference  structure  (Parnell  et  al.,  2013). 
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Step  3:  Develop  Evaluation  Measures 

The  final  step  in  the  hierarchy  development  is  to  establish  evaluation  measures  for 
the  lowest  level  objectives.  The  goal  of  this  step  is  to  establish  a  quantitative  measure, 
also  referred  to  as  an  attribute,  which  can  reflect  the  value  associated  with  a  particular 
objective.  This  measure  can  then  be  mathematically  associated  with  a  value  score 
through  the  use  of  a  CVF  which  is  created  in  Step  4.  The  value  measures  are  categorized 
into  the  four  types  shown  in  Table  4,  in  order  of  preference  (Keeney,  1992).  Regardless 
of  the  value  measure  type  used,  the  scale  should  be  clear  and  meaningful  to  the  SDM. 

The  end  points  of  the  scale  should  be  chosen  such  that  they  are  likely  to  be  inclusive  of 
any  future  alternative  (Von  Winterfeldt  &  Edwards,  1986).  The  end  points  are  used  in 
the  construction  of  CVFs.  If  a  future  alternative  was  presented  that  did  not  fall  within 


Table  4:  Types  of  Value  Measures  (Parnell  et  al.,  2013) 


Preference 

Type 

Measurement 

Method 

Description 

1 

Natural 

Direct 

The  measure  is  commonly  interpreted  and 
directly  measures  the  subject  objective. 

2 

Constructed 

Direct 

The  measure  was  either  constructed 
specifically  for  the  hierarchy  or  must  be 
explained  and  directly  measures  the 
objective. 

3 

Natural 

Proxy 

The  measure  is  commonly  interpreted  and 
does  not  directly  measure  the  objective. 

4 

Constructed 

Proxy 

The  measure  was  either  constructed 
specifically  for  the  hierarchy  or  must  be 
explained  and  does  not  directly  measure  the 
objective. 

the  bounds  of  the  scale  it  could  not  be  compared  to  existing  alternatives  without 
reworking  multiattribute  value  function  and  re-assessing  all  previously  assessed 
alternatives. 
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Step  4:  Create  Value  Functions 

After  evaluation  measures  and  appropriate  scales  have  been  determined,  the 
decision  maker,  or  facilitator,  must  develop  CVFs  for  each  measure.  This  section 
examines  qualitative  characteristics/implications  and  techniques  used  in  the  establishment 
of  CVFs. 

Value  measures  have  two  attributes  that  should  be  considered  before  creating  the 
CVFs.  First,  a  value  measure  can  be  either  continuous  or  discrete.  Second,  a  value 
measure  can  be  monotonically  increasing,  monotonically  decreasing,  or  non-monotonic. 
The  monotonically  increasing  (decreasing)  case  has  increased  (decreased)  value  as  the 
measure  increases.  In  the  non-monotonic  case,  value  rises  then  falls  as  the  measure 
increases.  This  phenomenon  typically  reveals  a  merger  of  conflicting  values  and  can  be 
resolved  through  further  examination  of  the  value  hierarchy.  The  purpose  of  the  CVF  is 
to  convert  the  scale  for  each  value  into  a  common  value  scale.  When  preferential 
independence  exists  between  values,  the  CVF  is  referred  to  as  an  SAVF. 

There  are  four  primary  shapes  of  SAVFs  for  continuous  value  measures:  Linear, 

Concave,  Convex,  S-Curve.  A  decision  maker’s  value  preferences  may  be  different  for 

each  objective  (Parnell  et  al.,  2013).  The  SAVFs  are  constructed  in  accordance  with  the 

inputs  of  the  decision  maker  and  allow  for  the  model  to  precisely  match  the  preference 

structure  (Keeney,  1992)(Keeney  &  Raiffa,  1993).  The  implications  of  the  four 

categories,  illustrated  in  Figure  4,  are  explained  based  on  an  assumption  of  a 

monotonically  increasing  value  measure.  The  linear  function  represents  a  constant 

valuation,  where  each  unit  of  the  value  measure  holds  the  same  amount  of  value  for  the 

SDM.  This  is  often  utilized  for  monetary  attributes.  The  concave  (convex)  function  has 
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decreasing  (increasing)  marginal  value,  where  each  unit  of  the  value  measure  holds  less 


(more)  value  as  it  approaches  the  maximum.  Concave  and  convex  functions  are  typically 
represented  with  an  exponential  curve  fitting  operation.  The  S -Curve  function  captures 
both  a  convex  and  concave  region.  This  is  typically  representative  of  a  value  measure 
with  an  optimal  point  or  goal.  In  the  case  of  a  monotonically  decreasing  measure  the 
implications  of  each  function  are  simply  reversed.  When  the  value  measure  has  a  small 


Figure  4:  Example  SAVFs  (Parnell  et  al.,  2013,  p.  196) 

number  of  discrete  levels  a  categorical  value  function,  illustrated  in  Figure  5,  can  be 
employed  (Parnell  et  al.,  2013).  The  SAVFs  described  have  all  hinged  on  the  assumption 
of  preferential  independence,  where  the  value  of  a  single  attribute  is  not  dependent  on  the 
value  of  any  other  single  attribute  in  the  value  structure.  When  this  assumption  does  not 
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hold  the  facilitator  must  account  for  interaction  between  the  attributes.  This  can  be 


captured  utilizing  the  multilinear  functional  form  shown  in  equation  (3)  (Keeney  & 
Raiffa,  1993). 


Figure  5:  Categorical  Single  Attribute  Value  Function 


n  n 

v  ( x ) = s!  ■  vi  (xi ) + Z  X  sij  •  vi  (x< )  •  vj  (xj  ) 

i=l  i= 1  j> 1 

Where 

X  =  The  interacting  attributes 
=  Scaling  Constant  for  attribute  i 
v;  (x )  =  S  AVF  for  attribute  ?' 

x.  =  Scaling  Constant  for  the  biteraction  between  attribute  i  and  j 


CVFs  are  constructed  utilizing  either  silver  or  platinum  inputs,  as  described 

above,  obtained  through  one  of  three  elicitation  techniques;  direct  rating,  difference 
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standard  sequence,  and  bisection.  The  fundamental  goal  of  the  elicitation  process  is  to 
attain  enough  information  from  the  SDM  to  characterize  the  value  space  and  check  for 
consistency  (Von  Winterfeldt  &  Edwards,  1986). 

Direct  rating  is  a  numerical  estimation  technique  employed  when  there  are  a  small 
number  of  discrete  levels  in  the  value  measure  or  when  there  is  small  set  alternatives 
under  consideration  and  the  SDM  can  make  firm  judgment  about  preferences  between 
levels.  The  SDM  is  first  asked  to  identify  the  most  preferred  and  least  preferred  level. 

The  most  preferred  level  is  assigned  a  value  score  of  1  and  the  least  preferred  level  is 
assigned  a  value  score  of  0.  The  SDM  is  then  asked  to  score  the  intermediate  levels 
based  on  strength  of  preference.  Once  complete,  consistency  is  checked  by  examining 
and  confirming  the  order  and  relative  differences  between  levels.  For  example,  one  may 
ask  if  the  value  difference  between  level  1  and  2  is  truly  larger,  or  smaller,  than  the  value 
difference  between  level  2  and  3.  The  data  can  then  be  used  to  construct  a  categorical 
SAVF  (Von  Winterfeldt  &  Edwards,  1986). 

The  difference  standard  sequence  technique  is  an  indifference  method  utilized 
when  the  value  measure  is  either  continuous  or  includes  a  large  number  of  discrete  levels. 
The  SDM  is  first  asked  to  identify  a  least  preferred  level,  xo  ,  and  an  initial  interval,  A, 

,that  is  approximately  one-fifth  to  one-tenth  of  the  overall  range.  Then  values,  v(xa )  =  0 
and  v(xa  +  A,)  =  v(x, )  =  1,  are  assigned  arbitrarily.  The  SDM  is  then  asked  to  determine 
additional  A  values  such  that  v(x  2  -  x ,)  =  v(x3  -  x2)  =  ..  .v(xi  +  xi  _, ) ,  where  A  =  xt  -  xt_  ,  . 
This  procedure  is  repeated  until  the  most  preferred  level  is  reached.  The  delta  values  are 
then  summed  and  normalized  between  0  and  1.  This  data  obtained  can  be  used  to  create  a 
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piecewise  linear  function  or  to  conduct  a  curve  fitting  operation.  In  either  case  the 
function  provides  a  means  to  convert  the  native  value  measure  into  units  of  value  through 
interpolation.  Consistency  is  obtained  through  the  granularity  of  the  initial  interval  size 
that  is  chosen.  If  the  interval  is  too  large  the  elicitation  may  be  very  short,  however  the 
interpolation  of  values  not  directly  assessed  may  not  be  accurate.  Conversely,  if  the 
interval  is  too  small  the  elicitation  may  be  overly  burdensome  but  would  obtain  more 
accurate  results  (Von  Winterfeldt  &  Edwards,  1986). 

The  bisection  technique  is  another  indifference  method  utilized  when  the  value 
measure  is  continuous  or  includes  a  large  number  of  discrete  levels.  In  this  technique  the 
most,  x  ,  and  least,  x0  ,  preferred  levels  of  the  value  measure  are  determined.  Again, 

v(xc)  =0  and  v(x  )  =  1  are  assigned  arbitrarily.  The  SDM  is  then  asked  to  determine  a 
level,  x5 ,  that  obtains  a  value,  v(x5)  =  .5  .  This  midpoint  value  can  then  be  used  to  fit  an 
exponential  approximation  to  the  data.  This  approximation  is  then  checked  for 
consistency  by  asking  the  SDM  to  further  subdivide  the  scale  to  obtain  an  *  25  and  x  75 

level  such  that  v(  x25  )  =  .25  and  v(x15 )  =  .75  .  The  elicitation  is  concluded  if  the 
additional  points  are  consistent  with  the  exponential  approximation.  If  not  consistent  the 
analyst  would  choose  a  different  functional  form  or  further  subdivide  the  scale.  The  goal 
of  this  technique  is  to  obtain  sufficient  data  to  allow  for  confident  interpolation  of  all 
other  value  scores  (Von  Winterfeldt  &  Edwards,  1986). 

Step  5:  Weight  Value  Hierarchy 

Establishing  weight  factors  for  the  value  hierarchy  is  the  final  step  in  development 
of  the  MAVF.  The  weights  are  utilized  to  assess  the  decision  maker’s  strength  of 


33 


preference  among  all  of  the  lower  level  objectives  contained  in  the  hierarchy.  There  are 
many  methods  employed  for  the  determination  of  weight  factors  including:  swing 
weighting,  rank  sum,  rank  exponent,  rank  reciprocal  and  rank-order  centroid  (ROC). 

Swing  weighting  or  “trade  offs”  (Buede,  2000)  is  an  indirect  subjective 
assessment  technique  in  which  the  SDM  ranks  the  swing  values  in  terms  of  contribution 
to  the  overall  value.  The  swing  value  is  the  value  obtained  from  swinging  a  specific 
measure  from  the  least  preferred  level  to  the  most  preferred  level.  The  highest  (lowest) 
ranked  value  is  assigned  it  an  arbitrary  score  of  100  (1).  The  SDM  then  determines  equal 
contribution  points  between  all  other  values  and  the  highest  (lowest)  ranked  values.  If 
swinging  the  2nd  value  measure  from  the  least  to  most  preferred  levels  provided  as  much 
value  as  swinging  the  1st  ranked  value  measure  from  the  least  preferred  level  to  a  70% 
level,  then  the  2nd  value  measure  would  have  a  weight  of  70.  This  process  is  repeated  for 
all  other  value  measures,  using  the  highest  (lowest)  ranked  value  as  an  anchor  to  which 
all  others  are  compared.  The  raw  weights  are  then  summed  and  normalized  between  zero 
and  one  (Buede,  2000).  This  technique  requires  direct  involvement  of  the  decision  maker 
or  decision  board  to  perform  ranking  and  weighting  determinations. 

Rank  sum,  rank  exponent,  rank  reciprocal  and  ROC  leverage  a  subjective 
assessment  of  the  swing  ranks  of  each  value  to  transform  the  order  into  swing  weights. 

The  SDM  is  asked  to  establish  a  rank  order  based  on  “the  relative  value  associated  with 
increasing  from  the  bottom  to  the  top  of  each  value  scale”  (Buede,  2000).  Equation  (4), 
(5),  (6),  and  (7)  show  the  mathematical  manipulations  required  to  turn  the  ranks  into 
weights  for  the  sum,  exponent,  reciprocal,  and  ROC  methods,  respectively  (Buede, 
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2000).  These  techniques  require  a  lower  level  of  involvement  from  the  SDM  and  thus 
can  be  valuable  when  access  is  limited. 


Rank  Sum 

f  \ 

k-r.+l 

w.  =  - - - 

vvi  k 

Xk-r.+i 

V  i= 1 

Where 

k  =  the  total  number  of  attributes 
Wj  =  the  value  weight 
r  =  the  value  rank 


(4) 


Rank  Exponent 

f  ) 

(k-r  +1)1 

w.-=  ~k - 1 - 

I ](k-rj  +  iy 

V  j=l 

Where 

k  =  the  total  number  of  attributes 
w;  =  the  value  weight 
r  =  the  value  rank 
z  =  measure  of  dispersion 


(5) 
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(6) 


Rank  Reciprocal 
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Where 

k  =  the  total  number  of  attributes 
w’(  =  the  value  weight 
r  =  the  value  rank 


Rank-Ordered  Centroid 
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Where 

k  =  the  total  number  of  attributes 
w’  =  the  value  weight 
rt  =  the  value  rank 

After  all  the  weight  factors  are  determined,  they  are  normalized  between  0  and  1. 
The  weights  are  then  multiplied  by  the  associated  CVFs  in  a  weighted  additive  value 
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function.  The  final  MAVF  is  utilized  to  calculate  the  value  score  for  each  candidate 
alternative.  The  scores  can  then  be  ranked  or  compared  on  a  common  scale,  allowing  the 
decision  maker  to  make  an  informed  decision  (Kirkwood,  1996). 

Step  6:  Alternative  Generation 

In  the  VFT  methodology  the  value  structure  of  the  decision  maker  is  considered 
prior  to  determining  the  alternatives  for  consideration.  There  are  two  common  problems 
encountered  during  alternative  generation,  too  many  alternatives  or  too  few  alternatives. 
There  are  several  methods  highlighted  in  the  literature  to  aid  in  both  of  these  problems 
(Kirkwood,  1996).  In  the  event  of  too  many  alternatives,  the  decision  maker  can  utilize 
screening  criteria  to  place  firm  limits  on  specific  easy  to  obtain  data  in  an  effort  to  narrow 
the  field  of  potential  alternatives  (Keeney,  1992).  Additionally,  Kirkwood  (1996) 
identifies  the  concept  of  dominance,  where  “an  alternative,  at ,  dominates  a  second 
alternative,  a2 ,  if  at  is  at  least  as  preferred  as  a2  with  respect  to  all  the  attributes  and 
more  preferred  with  respect  to  at  least  one  attribute”  (p.  229).  An  alternative  that  is 
dominated  by  another  alternative,  based  on  the  concept  described  above,  can  be  removed 
from  consideration.  In  the  event  of  too  few  alternatives,  a  strategy  generation  table  can 
be  employed  to  stimulate  creative  alternative  generation.  The  strategy  generation  table 
lays  out  the  full  combinatorial  set  of  decisions  enabling  decision  maker  to  look  for 
strategies  that  had  previously  not  been  considered  (Kirkwood,  1996). 

Step  7:  Alternative  Scoring 

At  this  point  in  the  VFT  process,  the  decision  maker  has  a  fully  developed  model 
and  a  set  of  viable  alternatives  for  consideration.  To  obtain  value  scores  for  each 
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alternative,  the  decision  maker  must  assess  each  alternative  against  all  of  the  quantitative 
and  qualitative  value  measures  established  in  the  model.  In  some  cases  this  is  as  simple 
as  pulling  a  piece  of  available  data  such  as  square  footage  or  miles  per  gallon.  In  other 
cases  it  requires  the  decision  maker  to  utilize  a  constructed  scale  to  establish  the  score. 
After  all  of  the  alternatives  have  been  scored  for  each  of  the  value  measures,  a  composite 
value  score  can  be  calculated.  The  composite  value  score  allows  the  decision  maker  to 
compare  alternatives  on  a  best  overall  value  basis  or  compare  all  alternatives  to 
established  value  thresholds. 

Step  8:  Deterministic  Analysis 

Deterministic  Analysis  is  the  process  of  calculating  value  scores  from  the 
alternative  scoring  inputs.  To  compute  the  composite  value  score  for  each  alternative  the 
individual  value  measure  scores  and  associated  weight  factors  are  input  into  the  MAVF. 
The  value  scores  are  then  examined  to  determine  relevant  conclusions  and 
recommendations.  The  output  of  this  process  is  used  to  communicate  the  value 
judgments  to  the  decision  maker  in  Step  10  (Keeney,  1992). 

Step  9:  Sensitivity  Analysis 

The  sensitivity  analysis  is  completed  after  the  deterministic  analysis.  The  purpose 

of  the  sensitivity  analysis  is  to  “determine  the  impact  on  the  ranking  of  alternatives 

[from]  changes  in  various  model  assumptions”  (Kirkwood,  1996,  p.  82).  The  sensitivity 

analysis  provides  the  decision  maker  with  a  sense  of  model  strength  or  robustness.  For 

example,  sensitivity  analysis  could  identify  if  the  alternative  rankings  produced  by  the 

model  would  change  given  slight  variations  in  the  decision  maker’s  subjective  weight 

factor  determination.  If  this  were  the  case,  the  decision  maker  should  be  aware  of  this 

38 


sensitivity  and  ensure  that  the  weight  factors  were  accurate  before  proceeding  with  the 
recommended  alternative. 

Step  10:  Conclusions  &  Recommendations 

The  final  step  of  the  VFT  model  development  process  is  to  provide  conclusions 
and  recommendations  to  the  decision  maker.  This  is  a  fairly  straight  forward  concept.  In 
this  stage  the  analyst  would  gather  the  data  and  analysis  to  build  a  summary  level 
document  or  section  explaining  the  conclusions  of  the  decision  analysis  effort.  This 
document  or  section  would  include  recommendations  and  any  points  of  clarification  or 
sensitivities  of  the  model. 

Uncertainty  in  Multiattribute  Decision  Model 

At  the  beginning  of  Chapter  2,  it  was  stated  that  the  two  primary  contributors  that 
make  decisions  complex  are  uncertainty  and  multiple  conflicting  objectives  (Keeney  & 
Raiffa,  1993)(Von  Winterfeldt  &  Edwards,  1986).  Interface  decisions  with  multiple 
objectives  under  the  assumption  of  certainty  is  the  focus  of  this  research.  However,  the 
addition  of  uncertainty  could  enable  the  model  better  match  reality.  This  section  will 
discuss  important  terminology  and  then  explore  two  approaches  for  handling  uncertainty 
in  decision  scenarios  with  multiple  objectives.  Finally,  strengths  and  weaknesses  of  each 
approach  are  outlined. 

Understanding  the  difference  between  value  and  utility  is  a  critical  concept  to 
decision  analysis  practitioners.  The  terms  value  and  value  function  are  explained  in 
Table  3.  Table  5  captures  explanations  of  utility  and  utility  functions.  Matheson  and 
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Abbas  describe  two  primary  approaches  for  coalescing  a  decision  maker’s  trade-off 
preferences  and  risk  preferences  to  compare  alternatives  (2005).  The  approaches  are 
discussed  below. 

Table  5:  Utility  Theory  Terminology 


Term 

Description 

Utility 

Concerned  with  capturing  a  “decision  maker’s  attitude  toward  risk¬ 
taking”  (Parnell  et  al.,  2013,  p.  56). 

Utility 

Function 

“A  mapping  of  the  utility  metric  from  the  value  metric  in  the  case  of  a 
single-dimensional  utility  function  or  from  all  of  the  performance  scores 
in  the  case  of  a  multidimensional  utility  function”  (Parnell  et  al.,  2013,  p. 
56). 

Approach  1:  The  “Standford  School  approach”  (Matheson  &  Abbas,  2005,  p. 
229)  depicted  in  Figure  6,  leverages  a  multidimensional  value  function  to  capture  all 
trade-off  preferences  which  converts  all  component  value  measures  into  a  single  ‘value’ 
metric.  After  the  multidimensional  value  function  is  established,  a  single-dimensional 


Figure  6:  Approach  1  (Parnell  et  al.,  2013,  p.  59) 

utility  function,  capturing  the  decision  maker’s  risk  preferences  over  the  single  value 

metric,  can  be  constructed.  The  end  result  is  a  single  utility  metric  for  each  alternative 
(Parnell  et  al.,  2013). 
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Approach  2:  The  “Keeney-Raiffa  approach”  (Matheson  &  Abbas,  2005,  p.  229) 
depicted  in  Figure  7,  employs  a  Multidimensional  utility  function  which  captures  both 
trade-off  and  risk  preferences.  This  approach  leverages  the  assumption  or  verification  of 


Figure  7 :  Approach  2  (Parnell  et  al.,  2013,  p.  59) 

independence  conditions  to  simplify  the  functional  form  of  the  utility  function  and 

expedite  the  elicitation  process.  Like  Approach  1,  the  end  result  is  a  single  utility  metric 
for  each  alternative  that  can  be  used  for  objective  comparison  (Parnell  et  al.,  2013). 

Both  approaches  share  the  same  goal:  provide  objective  criteria  for  comparing 
alternatives  in  support  of  decision  making.  Approach  1  is  regarded  as  easier  to 
implement  because  the  bulk  of  the  elicitation  is  focused  on  developing  a  deterministic 
value  function.  Additionally,  if  the  SDM  is  risk  neutral  or  the  decision  is  low  risk  the 
second  step  can  be  removed  (Parnell  et  al.,  2013).  Finally,  Approach  1  captures  utility 
dependence  without  the  elicitation  complexity  associated  with  the  multidimensional 
utility  function  employed  by  Approach  2  (Matheson  &  Abbas,  2005).  A  weakness  of 
Approach  1  is  the  use  of  a  unit-less  value  metric  which  makes  intuitive  construction  of  a 
utility  function  over  the  value  metric  difficult  (Keeney  &  Raiffa,  1993).  Approach  2  has 
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the  benefit  of  a  single  step,  simple  utility  function,  and  a  straightforward  elicitation 
process  if  the  independence  conditions  are  verified  or  assumed  (Parnell  et  al.,  2013). 
However,  in  many  cases  the  independence  conditions  do  not  hold  and  making  the 
assumption  that  they  do  hold  ignores  potentially  important  interactions  (Matheson  & 
Abbas,  2005). 

Summary 

Chapter  2  provided  an  overview  of  literature  in  the  areas  of  open  systems  and 
decision  analysis.  First,  the  areas  of  MOSA  in  the  systems  engineering  process,  MOSA 
policy,  and  MOSA  guidance  were  described.  Next,  two  decision  analysis  methods,  the 
AHP  and  VFT,  were  examined  followed  by  a  detailed  exploration  of  the  execution  of  the 
VFT  decision  model  development  process.  Finally,  Chapter  2  concluded  with  an 
exploration  of  methods  of  incorporating  uncertainty  into  multiattribute  decision  models. 
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III.  Methodology 


Chapter  3  follows  the  VFT  process  discussed  in  Chapter  2,  beginning  with 
justification  of  VFT  selection  followed  by  discussion  of  problem  formulation.  Next,  the 
method  by  which  data  was  collected  and  aggregated  into  a  hierarchy  that  represents  the 
preference  structure  of  the  SDM  is  addressed.  Following  the  discussion  of  hierarchy 
construction,  this  chapter  describes  methods  by  which  value  measures,  value  functions, 
and  weight  factors  were  determined.  Finally,  the  results  of  a  hierarchy  quality  evaluation 
are  explained.  Chapter  2  indicates  several  additional  steps  beyond  value  hierarchy 
weighting.  Alternative  Generation,  Alternative  Scoring,  Deterministic  Analysis  and 
Sensitivity  Analysis  will  be  discussed  in  Chapter  4.  Conclusions  and  Recommendations 
are  captured  in  Chapter  5 . 

Methodology  Selection 

The  Medium  Altitude  UAS  System  Program  Office  has  the  challenge  of 

integrating  many  subsystems  into  the  Predator  and  Reaper  UAS.  This  requirement 

highlights  the  need  for  a  defensible,  repeatable,  and  objective  evaluation  framework  to 

support  making  IIM  decisions  while  considering  technical  and  programmatic  factors. 

The  program  office  considers  multiple  criteria  in  the  choice  of  IIMs  for  the  many 

interfaces  employed  in  a  UAS.  Each  new  interface  requires  a  new  evaluation  though  the 

quantity  of  potential  IIMs  is  small  and  the  relevant  evaluation  factors  remain  constant. 

This  research  is  focused  on  building  a  deterministic  evaluation  framework  to  ensure 

accurate  and  consistent  choices  that  reflect  the  SDMs  value  structure  while  not  requiring 

direct  involvement  in  every  evaluation.  VFT  was  chosen  over  AHP  because  literature 
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indicated  that  it  is  better  suited  for  repeated  decisions  and,  after  constructed,  VFT  models 
were  usable  in  the  absence  of  the  SDM  (Belton,  1986).  VFT  was  used  to  construct  an 
evaluation  framework  consistent  with  the  values  of  the  program  office  with  respect  to 
IIMs. 

Step  1:  Problem  Identification 

As  discussed  in  Chapter  2,  the  goal  of  the  problem  identification  step  is  to  gain  a 
complete  and  contextually  accurate  understanding  of  the  problem.  Complete 
understanding  of  the  IIM  problem  was  established  through  an  amalgamation  of  three 
sources:  the  author’s  personal  experiences,  DoD  acquisition  directives,  and  interviews 
with  SDMs  in  the  program  office.  The  author’s  experience  is  in  the  area  of  payload 
integration  with  UAS  showed  that  many  IIM  decisions  were  entrusted  to  the  contractor 
due  to  schedule  urgency.  Abdicating  responsibility  for  IIM  decisions  to  the  contractor 
resulted  in  closed  interfaces  that  required  constant  change  which  generated  challenges  for 
system  integration,  system  support  and  technology  evolution.  Research  in  interface 
integration  revealed  that  DoD  Directive  5000.01  states  “a  modular,  open-systems 
approach  shall  be  employed,  where  feasible”  (Office  of  the  Undersecretary  of  Defense 
for  Acquisition  Technology  &  Logistics,  2007).  Interviews  with  SDMs  from  the  program 
office  clarified  the  problem.  By  definition,  implementation  of  open  interfaces  is  always 
feasible,  but  it  isn’t  always  practical  or  reasonable.  The  issue  of  practicality  includes 
factors  other  than  technical  performance.  Contextual  elements  of  the  decision  frame  such 
as  schedule  urgency,  cost,  interface  utilization  and  classification  needed  to  be  considered 
when  determining  the  practicality  of  implementing  an  open  interface.  The  three  sources 
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listed  above  lead  to  the  conclusion  that  DoD  acquisition  lacks  a  defensible,  repeatable, 
and  objective  method  for  determining  if  implementing  a  modular,  open-systems  approach 
is  appropriate  for  a  specific  interface.  The  OSJTF  reinforced  this  finding  when  it  stated 
that  “key  interfaces  should  be  examined  very  carefully  to  insure  that  the  use  of  an  open 
standard  is  both  feasible  and  appropriate  based  on  performance  and  business  objectives” 
(2004). 

To  develop  the  MQ-l/MQ-9  value  structure  with  respect  to  IIMs,  Gold  and  Silver 
standard  sources  were  employed  resulting  in  a  set  of  76  decision  factors.  Table  6.  Gold 
standard  documentation  was  examined  to  gain  contextual  understanding  of  values  related 
to  open  interfaces.  The  OSJTF  Program  Manager’s  Guide  was  consulted  to  provide 
understanding  of  doctrinal  based  decision  factors.  Journal  articles  and  conference 
submittals  in  the  areas  of  open  system  integration  and  evolutionary  acquisition  were 
examined  for  additional  factors.  In  an  acquisition  program  office,  IPT  members,  in  the 
areas  of  Program  Management,  Engineering,  Logistics,  Operations,  Contracting  and 
Finance,  would  advise  the  SDM  on  program  decisions.  To  capture  this  dynamic, 
interviews  were  conducted  with  members  the  IPT.  The  research  and  interviews  resulted 
in  a  comprehensive  list  of  76  factors  providing  a  foundation  for  hierarchy  development. 
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Table  6:  Decision  Factor  List 


Decision  Factor 

Source 

1 

Mutability  of  the  connected  systems 

(Dillard  &  Ford, 
2007) 

2 

Logistics  support  plan 

3 

Time  criticality  of  future  iterations  /  development  of  tenant  systems 

4 

Cycle  time  between  phases  or  upgrades 

5 

Amount  of  interdependency  between  systems. . .  effect  of  one 
systems  evolution  on  another 

6 

Evolving  technologies 

(Ford  &  Dillard, 
2008) 

7 

Use  of  legacy  hardware 

8 

Need  for  flexibility  in  acquisition  to  manage  uncertainty  in 
technology 

9 

New  challenges  that  are  difficult  to  forecast 

10 

Need  for  interoperability 

11 

Integration  with  joint  services 

12 

Collaboration  with  allies 

13 

Need  for  rapid  acquisition  response 

14 

Evolving  Technologies 

(Ford  &  Dillard, 
2009) 

15 

Evolving  Programmatic  /  Acquisition  Environment 

16 

Changing  Acquisition  Environment 

17 

Continuous  improvement  required 

18 

Changing  threat  matrices 

19 

New  Challenges/Evolving  threats  that  are  difficult  to  forecast 

20 

Dynamic  Threat 

21 

Ability  to  disclose  design  information 

22 

Joint/allied  interactions  with  the  system  interface 

23 

Constrained  funding 

24 

Little  flexibility  in  money  for  development 

25 

Short  capability  improvement  cycle  times 

26 

System  in  urgent  need  of  improvement 

27 

Requirement  for  rapid  evolutionary  improvement 

28 

Little  flexibility  in  time  for  development 

29 

Rate  of  iteration 
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30 

Leverage  of  COTS  components 

(OSJTF,  2004) 

31 

Potential  for  technology  obsolescence 

32 

Changing  Requirements 

33 

Use  of  common  or  cross  platform  components 

34 

Development  urgency  -  need  for  reduced  development  timelines 

35 

Multiple  sources  of  supply  for  a  host  or  tenant 

36 

Competition  between  vendors 

37 

Frequency  of  configuration  changes  in  tenant  systems  (Stability  of 
Tenant  Systems) 

(IPT  Engineering, 
personal 
communication, 
September  12, 
2013) 

38 

Frequency  of  configuration  changes  in  host  system  (Stability  of 

Host  System) 

39 

Volatility/Stability  of  industry  use  of  the  open  standard 

40 

Maturity  of  the  open  standard 

41 

Availability  of  open  standards 

42 

Security  classification  of  interface  description  or  mechanism 

43 

Users  of  the  interface  (Quantity) 

44 

Number  of  instances  of  the  interface  on  a  given  platform 

45 

Sources  of  tenants 

46 

Predictability  of  changes  at  the  interface 

47 

Complexity  of  the  Interface 

48 

Will  the  interface  decision  restrict  future  competition 

(IPT  Contracting, 
personal 
communication, 
September  13, 
2013) 

49 

Frequency  of  system  changes  at  the  interface 

(IPT  Program 

Management, 

personal 

communication, 

September  13, 

2013) 

50 

Interface  proliferation  (the  use  of  the  interface  on  coupled  systems) 

51 

The  number  of  open  standards  that  exist  that  could  do  the  job 

52 

Proprietary  nature  of  ultra  high  performance  systems 

53 

Level  of  system  security  required 

54 

Number  of  users  of  the  interface 

55 

Capability  (Training  Level)  of  the  maintenance  crew 

56 

Time  available  to  perform  a  repair  and  replace 

57 

Maintenance  Environment 

58 

Amount  of  change  and  availability  of  funding 

59 

Interfacing  experience  of  the  integrator 

60 

Urgency  of  capability  implementation  as  a  function  of  time  required 
to  implement  an  open  interface 
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61 

Change  to  coupled  systems 

(IPT  Fogistics, 
personal 
communication, 
September  17, 
2013) 

62 

Training  level  of  personnel 

63 

Maintenance  concept 

64 

Urgency  of  repair 

65 

Frequency  remove/replace  at  the  interface 

66 

System  stability  (effect  of  technological  change) 

(IPT  Finance, 
personal 
communication, 
September  19, 
2013) 

67 

Stability  of  the  intended  interface  standard 

68 

Ease  of  maintenance  desired/required 

69 

Desired/required  training  level  for  maintenance  Personnel 

70 

Cost  of  development/implementation  of  the  standard 

71 

Competition 

72 

Failure  rate  of  the  interface 

73 

Mission  disparity 

(IPT  Operations, 
personal 
communication, 
October  3,  2013) 

74 

Security  level  of  associated  equipment 

75 

Urgency  of  development 

76 

Complexity  of  system  integration 

Step  2:  Create  Value  Hierarchy 

The  full  factor  list  found  during  problem  identification  was  used  in  an  affinity 
diagram,  aggregation  and  sorting  exercise,  using  Microsoft  Excel,  to  group  and 
categorize  similar  factors.  The  factors  were  then  organized  into  in  a  comprehensive 
hierarchy,  Figure  8.  Platinum  standard  inputs  were  used  to  confirm  and  refine  the 
hierarchy. 
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Figure  8:  IEF  Value  Hierarchy 


Value  Hierarchy  Description 

The  strategic  objective  of  the  hierarchy  was  to  maximize  the  value  of  an  open 

interface  implementation.  This  strategic  objective  was  decomposed  into  three 

fundamental  objectives:  Minimize  Acquisition  Cost,  Meet  Schedule  Expectations,  and 

Meet  Acquisition  Performance  Expectations.  Minimize  Acquisition  Cost  assesses  the 

cost  of  implementing  open  as  compared  to  closed  with  consideration  of  the  number 

integrations  at  the  interface.  Meet  Schedule  Expectations  addresses  the  issue  of  schedule 
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pressure  for  capability  integration  at  the  interface.  Meet  Acquisition  Performance 
Expectations  considers  those  interface  performance  and  contextual  factors  other  than  cost 
and  schedule.  The  following  are  descriptions  of  lowest  level  values  within  the  meet 
acquisition  performance  expectations  branch: 

Adjust  to  Change:  Captures  the  interface  performance  and  contextual  factors 
related  to  change.  The  value  of  an  open  interface  is  related  to  both  the  maturity  of  the 
interface  and  the  amount  of  change  that  will  occur  at  the  interface.  This  value  is  broken 
into  three  sub  values: 

Adjust  to  Technology  Change :  This  value  addresses  the  volume  of  change 
driven  by  technology  alteration  or  maturation. 

Adjust  to  Threat  Change :  This  value  addresses  the  volume  of  change 
driven  by  changing  adversary  tactics. 

Minimize  Interface  Change :  This  value  refers  to  the  maturity  of  the 
interface  selected.  If  the  interface  option  that  is  available  is  of  a  low  maturity  level  it 
would  be  of  limited  value  to  the  program  because  it  would  likely  require  modification. 

Protect  Information:  Captures  the  need  to  protect  information  about  the 
capabilities  of  connected  systems  at  the  appropriate  level.  Making  an  interface  open 
implies  a  willingness  to  share  information  about  the  interface.  This  sharing  could 
inadvertently  provide  information  about  the  capability  of  a  system  connected  at  the 
interface.  Thus,  highly  protected  systems  would  limit  the  value  of  openness. 

Support  Users:  Captures  the  interface  performance  factors  associated  with 
ensuring  the  organizations  and  systems  that  utilize  the  interface  are  supported.  This  value 
is  broken  into  three  sub  values: 
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Support  User  Community:  This  value  addresses  the  need  of  the  interface 


to  support  the  using  community.  A  highly  varied  using  community  would  increase  the 
value  of  an  open  interface. 

Support  Quantity  of  Systems:  This  value  addresses  the  need  of  the 
interface  to  support  multiple  functionally  equivalent  systems.  For  the  purposes  of  this 
research,  two  systems  are  deemed  functionally  equivalent  if  they  are  used  to  meet  the 
same  requirement.  It  is  not  the  performance  of  the  systems  that  is  compared  but  the 
requirements  to  which  they  are  held. 

Support  Variety  of  Systems:  This  value  addresses  the  need  of  the  interface 
to  support  multiple  functionally  different  systems.  For  the  purposes  of  this  research  two 
systems  are  functionally  different  if  they  are  used  to  meet  different  requirements. 

Step  3  and  4:  Evaluation  Measures  and  Value  Functions 

Upon  completion  of  the  value  hierarchy,  evaluation  measures  were  established 
using  silver  and  for  each  of  the  lowest  level  hierarchical  elements.  The  goal  is  to  provide 
a  means  of  objectively  measuring  each  alternative  against  all  of  the  values  using  scales 
that  are  easily  understood  by  the  intended  audience.  Silver  and  platinum  standard  inputs 
were  used  to  choose  value  measures  and  contstruct  the  component  value  functions.  The 
measures  employed  were  both  direct  and  proxy  using  both  natural  and  constructed  scales. 
After  the  measures  were  established,  value  functions  were  developed  for  each  value  to 
establish  a  single  common  scale.  Categorical,  exponential,  piecewise  linear  and 
multilinear  value  functions  were  employed.  All  evaluation  measures  and  value  functions 
are  described  below. 
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Adjust  to  Technology  Change :  This  value  is  measured  using  a  negatively 
oriented,  categorical,  natural  proxy  scale  of  the  Technology  Readiness  Level  (TRL)  of 
connected  systems  to  assess  the  amount  of  technology  change  that  will  occur  at  the 
interface.  Because  more  than  one  system  can  be  connected  to  an  interface  an  average  of 
the  TRLs  of  the  connected  systems  is  utilized.  The  use  of  an  average  captures 
intermediate  levels  of  TRL  rather  than  applying  a  rounded  score.  While  this  more 
accurately  captures  the  amount  of  change  at  the  interface  it  requires  the  use  of  continuous 
function  rather  than  a  categorical  function.  There  is  an  inverse  relationship  between  TRL 
and  technology  change.  A  high  TRL  indicates  the  systems  connected  at  the  interface  are 
very  mature  and  thus  would  have  little  technology  change.  The  maximum  value  for  an 
open  interface  is  associated  with  a  TRL  5  while  TRL  9  receives  the  lowest  value.  The 
TRL  scale  is  shown  in  Table  7.  TRLs  1  through  4  were  not  included  because  technology 
of  this  maturity  would  not  be  considered  for  integration. 
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Table  7:  Technology  Readiness  Level  (Xi)(Assistant  Secretary  of  Defense  for 
Research  and  Engineering  (ASD  (R&E)),  2011) 


TRL  5 

"Fidelity  of  breadboard  technology  increases  significantly.  The  basic  technological 
components  are  integrated  with  reasonably  realistic  supporting  elements  so  they  can 
be  tested  in  a  simulated  environment.  Examples  include  "high-fidelity"  laboratory 
integration  of  components." 

TRL  6 

"Representative  model  or  prototype  system,  which  is  well  beyond  that  of  TRL  5,  is 
tested  in  a  relevant  environment.  Represents  a  major  step  up  in  a  technology's 
demonstrated  readiness.  Examples  include  testing  a  prototype  in  a  highfidelity 
laboratory  environment  or  in  a  simulated  operational  environment." 

TRL  7 

"Prototype  near  or  at  planned  operational  system.  Represents  a  major  step  up  from 

TRL  6  by  requiring  demonstration  of  an  actual  system  prototype  in  an  operational 
environment  (e.g.,  in  an  air-craft,  in  a  vehicle,  or  in  space)." 

TRL  8 

"Technology  has  been  proven  to  work  in  its  final  form  and  under  expected  conditions. 

In  almost  all  cases,  this  TRL  represents  the  end  of  true  system  development.  Examples 
include  developmental  test  and  evaluation  (DT&E)  of  the  system  in  its  intended 
weapon  system  to  determine  if  it  meets  design  specifications." 

TRL  9 

"Actual  application  of  the  technology  in  its  final  form  and  under  mission  conditions, 
such  as  those  encountered  in  operational  test  and  evaluation  (OT&E).  Examples 
include  using  the  system  under  operational  mission  conditions." 

Technology  change  was  represented  by  the  piecewise  linear  function  captured  with 
equation  (8)  and  shown  in  Figure  9.  The  piecewise  linear  function  was  chosen  because  it 
was  able  to  capture  an  inflection  point  in  the  value  function  at  average  TRL  7.2  described 
by  the  decision  maker. 


-0.1-Xj  +0.9 


vi(^)  =  i 


-0.5 -v,  +4.1 
-0.333  •  Xy  +  2.9 


[-0.1-jq+1.5 


for  8  <  JCj  <  9 
for  7.2<Xj<8 
for  6<.q<7.2 
for  5  <  v,  <  6 


(8) 
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Figure  9:  Adjust  to  Technology  Change  Value  Function 

Adjust  to  Threat  Change :  This  value  employs  a  positively  oriented  constructed 
direct  scale  measuring  the  threat  environment  to  assess  the  amount  of  change  that  will 
occur  at  the  interface  due  to  changing  adversary  tactics.  Multiple  adversaries  with 
changing  tactics  drive  a  high  level  of  change  which  results  in  a  high  value  for  an  open 
interface.  The  lowest  value  for  an  open  interface  is  associated  peace  time  which  is 
associated  with  the  most  consistent  tactics.  The  threat  environment  scale  is  shown  in 
Table  8. 
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Table  8:  Threat  Environment  (X2) 


5 

Multiple  Adversaries  w/  Changing  Tactics 

4 

Single  Adversary  w/  Changing  Tactics 

3 

Multiple  Adversaries  w/  Consistent  Tactics 

2 

Single  Adversary  w/  Consistent  Tactics 

1 

Peace  Time 

Threat  change  was  represented  by  the  categorical  function  captured  with  equation  (9)  and 
shown  in  Figure  10.  The  categorical  function  was  chosen  because  there  were  a  small 
number  of  discrete  levels  in  the  scale  enabling  the  use  of  direct  assessment  by  the 
decision  maker. 


0 

0.2 

v2  (jj  )  =|  0.75 
0.75 
TO 


for 

x2  =  1 

for 

x2  =  2 

for 

x2  =  3 

for 

x2  =  4 

for 

x2  =5 

(9) 


Figure  10:  Adjust  to  Threat  Change  Value  Function 

Minimize  Interface  Change :  This  value  is  assessed  with  a  positively  oriented  constructed 
direct  scale  measuring  the  maximum  maturity  of  the  interfaces  available  for 
implementation.  A  well-documented  interface  where  change  occurs  in  a  controlled 
manner  is  of  the  most  value.  Conversely,  an  undocumented  interface  where  change 
occurs  at  the  discretion  of  a  single  organization  or  integrator  is  of  the  least  value.  The 
threat  environment  scale  is  shown  in  Table  9.  Interface  change  was  represented  by  the 
categorical  function  captured  with  equation  (10)  and  shown  in  Figure  10.  The  categorical 
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function  was  chosen  because  there  were  a  small  number  of  discrete  levels  in  the  scale 
enabling  the  use  of  direct  assessment  by  the  decision  maker. 


Table  9:  Interface  Maturity  Level  (IML)  (X3) 


4 

Standards  Exist  and  are  documented  by  a  standards 
management  organization.  Change  occurs  in  a 
controlled  manner,  with  rigorous  review  and  community 
approval 

3 

Interface  is  documented  by  a  program  office  through  an 
interface  control  document  or  equivalent.  Change  is 
controlled  by  a  program  office 

2 

There  are  no  formally  documented  standards,  however 
there  is  industry  agreement.  Change  occurs  through 
industry  consensus. 

1 

There  is  no  defined  standard  and  there  appears  to  be  no 
agreement  among  integrators.  Changes  are  dictated  by 
each  integrator. 

0 

for 

x3  =  1 

0.15 

for 

x3  =2 

0.5 

for 

x3  =  3 

li.o 

for 

x3  =  4 

(10) 
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Level  1  Level  2  Level  3  Level  4 


Interface  Maturity  Level  (X3) 

Figure  11:  Minimize  Interface  Change  Value  Function 

Protect  Information :  This  value  is  measured  using  a  negatively  oriented  natural 

direct  scale  assessing  the  maximum  information  protection  level  (IPL)  required  for 
connected  systems.  Utilization  of  an  open  interface  implies  that  a  willingness  to  share 
information  about  the  interface  exists.  An  inverse  relationship  between  the  IPL  and  the 
value  of  an  open  interface  exists.  As  the  IPL  of  connected  systems  increases  the 
willingness  to  share  and,  subsequently,  the  value  of  an  open  interface  decreases.  The 
maximum  value  for  an  open  interface  is  associated  with  an  unclassified  IPL.  The 
minimum  value  is  associated  with  a  compartmentalized  top  secret  IPL.  The  IPL  scale  is 
shown  in  Table  10.  Information  protection  was  represented  by  the  categorical  function 
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Table  10:  Information  Protection  Level  (X4) 


_5 

Compartmentalized  TS 

4 

Top  Secret 

_3 

Secret 

2 

Controlled  Unclassified 

1 

Unclassified 

captured  with  equation  (11)  and  shown  in  Figure  12  below.  The  categorical  function  was 
chosen  because  there  were  a  small  number  of  discrete  levels  in  the  scale  enabling  the  use 
of  direct  assessment  by  the  decision  maker. 
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Information  Protection  Level  (X4) 


Figure  12:  Protect  Information  Value  Function 


Support  User  Community.  This  value  employs  a  positively  oriented  constructed  direct 
scale  assessing  the  user  community  that  will  interact  with  the  interface.  As  the  diversity 
of  the  user  community  increases  the  value  of  an  open  interface  increases.  The  maximum 
value  is  associated  with  a  multi-national  community  with  no  limitations  on  sharing. 
Conversely  a  single  unit  user  community  receives  the  least  value.  The  user  community 
scale  is  shown  in  Table  11.  User  community  support  was  represented  by  the  categorical 
function  captured  with  equation  (12)  and  shown  in  Figure  12.  The  categorical  function 
was  chosen  because  there  were  a  small  number  of  discrete  levels  in  the  scale  enabling  the 
use  of  direct  assessment  by  the  decision  maker. 
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Table  11:  User  Community  of  Connected  Systems  (X5) 


6 

Multi-Nation  (Not  Limited) 

5 

Multi-Nation  (Allied  Only) 

4 

Multi-Service 

3 

Single  Service 

2 

Single  MAJCOM 

1 

Single  Unit 

1.0 

for 

li 

ON 

0.65 

for 

*5=5 

0.5 

for 

II 

f-T 

0.25 

for 

x5  =  3 

0.2 

for 

*5=2 

0 

for 

x5  =  1 

(12) 


Single  Unit  Single  MAJCOM  Single  Service  Multi-Service  Multi-Nation  Multi-Nation 

(Allied  Only)  (NL) 

User  Community  of  Connected  Systems  (X5) 


Figure  13:  Support  User  Community  Value  Function 
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Support  Quantity  of  Systems:  This  value  is  assessed  with  a  positively  oriented 
natural  direct  scale  measuring  the  quantity  of  functionally  equivalent  systems  that 
connect  at  the  interface.  The  MQ-1  and  MQ-9  employ  both  small  quantity  high  value 
systems  as  well  as  readily  available  commercial  off-the-shelf  (COTS).  The  COTS 
systems  drive  the  maximum  for  this  scale.  A  use  case  was  developed  for  a  monitor 
installed  in  the  ground  station  with  the  requirements  of  22in  diagonal  viewing  angle,  and 
aspect  ratio  of  16:9.  This  resulted  in  38  functionally  equivalent  systems.  To  establish  the 
maximum  of  50,  -32%  was  added  to  the  number  found  in  the  use  case  to  provide  margin 
for  other  COTS  systems.  The  small  quantity  high  value  systems  drive  the  minimum  of 
one  for  this  scale.  It  is  common  that  a  system  is  developed  specifically  to  meet  a  set  of 
requirements  and  thus  only  1  functionally  equivalent  system  exists.  The  quantity  of 
systems  was  represented  by  an  exponential  function  captured  with  equation  (13).  The 
bisection  method  was  utilized  to  elicit  information  shown  in  Table  12.  The  exponential 
function  was  then  fit  to  those  points  as  shown  in  Figure  14. 


1 

^  _  e-0.25ix6-l) 
V6  (*6  )  —  ‘  i  _  ^-0.25  (50-1) 

0 


for  x6  >50 
for  1  <  x6  <  50 
for  x6<l 


(13) 
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Table  12:  Quantity  of  Systems  Elicited  Information  (X6) 


Value 

Attribute  Measure 

* 

1 

50 

v75 

0.75 

8 

x'5 

0.5 

3 

x'25 

0.25 

2 

x° 

0 

1 

Figure  14:  Support  Quantity  of  Systems  Value  Function 
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Support  Variety  of  Systems:  This  value  is  measured  using  a  positively  oriented 
natural  direct  scale  measuring  the  quantity  of  functionally  different  systems  that  connect 
at  the  interface.  The  F-16  external  storage  mechanical  interface  was  examined  to 
establish  a  reasonable  maximum  for  the  scale.  This  aircraft  was  chosen  because  it  shares 
a  similar  mission  to  the  MQ-1  and  MQ-9  and  has  been  in  service  much  longer.  It  is 
assumed  that,  because  of  the  service  longevity,  the  number  of  functionally  different 
connected  systems  has  reached  a  maximum.  There  were  three  categories  of  systems  that 
connected  to  the  interface:  munitions,  podded  sensors,  and  fuel  tanks.  There  were  seven 
different  munitions  connected  at  the  interface  (F-16  Armament).  Six  podded  sensor  types 
were  identified  by  the  decision  maker.  A  single  fuel  tank  was  assumed  by  the  author. 

This  accounted  for  14  functionally  different  connected  systems.  To  establish  the 
maximum  of  20,  -43%  was  added  to  account  for  classified  integrations.  The  minimum 
was  established  as  one  because  there  must  be  at  least  one  connected  system  for  an 
interface  to  exist.  The  variety  of  systems  was  represented  by  an  exponential  function 
captured  with  equation  (14).  The  bisection  method  was  utilized  to  obtain  the  information 
shown  in  Table  13.  The  exponential  function  was  then  fit  to  those  points  as  shown  in 
Figure  15. 


1 

l_e~0.162Hx7-l) 

V7  ( X1  )  ~  -0.1621  (20-1) 
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Table  13:  Variety  of  System  Elicited  Information  (X7) 


Value 

Attribute  Measure 

* 

1 

20 

v75 

0.75 

8 

x'5 

0.5 

5 

x'25 

0.25 

3 

x° 

0 

1 

Figure  15:  Support  Variety  of  Systems  Value  Function 
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Minimize  acquisition  cost :  This  value  employs  a  multilinear  value  function  with  a 
complementary  relationship  between  attributes.  The  multilinear  value  function  is 
employed  to  capture  an  interaction  between  the  cost  of  implementing  an  open  interface 
and  the  number  of  integrations  for  which  that  interface  will  be  used.  The  two  scales 
employed  to  construct  the  multilinear  function  were  the  cost  differential  (Xg)  between  an 
open  and  closed  interface  implementation,  and  the  number  of  integrations  (X9)  over  the 
planning  horizon.  Measuring  cost  differential  as  a  ratio  was  chosen  over  measuring  cost 
directly  because  it  avoided  issues  with  time  value  of  money  and  allowed  for  direct 
comparison  between  interface  implementations  regardless  of  acquisition  cost.  The 
maximum  of  an  open  interface  implementation  cost  equal  to  double  the  closed  interface 
implementation  cost  was  chosen  based  on  recommendation  by  the  SDM.  The  number  of 
integrations  employs  a  positively  oriented  natural  direct  scale.  The  maximum  was 
established  based  on  two  integrations  at  the  interface  per  year  through  the  10  year 
planning  horizon  for  a  total  of  20  integrations.  It  is  assumed  that  the  interface  exists  and 
thus  has  one  connected  system.  The  minimum,  zero  integrations,  indicates  that  no  new 
systems  will  be  integrated  to  the  interface  over  the  planning  horizon.  The  multilinear 
representation  of  minimizing  acquisition  cost  is  shown  in  equation  (15).  To  simplify  the 
elicitation,  linear  conditional  value  functions  scaled  between  0  and  1  were  assumed.  The 
scaling  constants  s8,  s9 ,  and  sHij  were  determined  through  structured  discussions  using  an 
Excel-based  iso-preference  tool  (Robbins,  2013).  As  can  be  seen  in  Figure  16,  a  series  of 
points  were  determined  throughout  the  two  attribute  value  space  to  support  scaling 
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constant  determination.  The  maximum  value  of  one  is  achieved  when  there  are  20 
integrations  and  a  zero  cost  differential.  The  lines  separating  colors  indicate 


(15) 


V89  U8 ,  *9 )  =  S8  •  V8  U8 )  +  s9  •  V9  09  )  +  s89  •  v8  (xg )  •  V9  (x9 ) 


Where 

Cost  Differential:  v8  (x8 )  =  1  -  x8 


0 

01  Cost  -  Cl  Cost 
Cl  Cost 

1 


for  01  Cost  <  Cl  Cost 

for  Cl  Cost  <  01  Cost  <  2  •  Cl  Cost 

for  01  Cost  >  2  •  Cl  Cost 


01  Cost=The  Cost  to  Implement  an  Open  Interface 
Cl  Cost=The  Cost  to  Implement  a  Closed  Interface 
Lower  cost  differential  is  preferred 


And 

Xq 

#  of  Integrations :  v9  (xg )  =  ^ 

indifference.  Any  point  along  the  indifference  line  indicated  indifference  for  the  decision 

maker.  The  shape  of  the  iso-preference  curves  are  driven  by  the  scaling  constants, 

determined  through  the  iterative  process  shown  in  Figure  17,  which  were  applied  to  the 

value  function.  This  process  was  repeated  until  the  preferences  depicted  in  the  tool  were 

consistent  with  the  preferences  of  the  SDM.  The  multilinear  value  function  for 

minimizing  acquisition  cost  is  shown  in  equation  (16). 
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Figure  16:  Minimize  Acquisition  Cost  Iso-Preference  Curves 


vg9(xg,  xg)  =  0.0  -(l  —  x8) +  0.05  • 


(16) 
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Yes 

1 

5)  Scaling  Constants 
Determined 


Figure  17:  Iterative  Multilinear  Scaling  Constant  Determination  Process 


Meet  schedule  expectations :  This  value  employs  a  multilinear  value  function  with 
a  complementary  relationship  between  attributes.  The  multilinear  value  function  was 
employed  to  capture  an  interaction  between  schedule  urgency  (Xio)  and  the  number  of 
integrations  (Xu)  for  which  that  interface  will  be  used.  Schedule  urgency  was  measured 
using  a  positively  oriented  constructed  proxy  scale  shown  in  Table  14,  which  measures 


Table  14:  Schedule  Urgency  of  Integrations  (Xio) 


6 

Nationally  Driven 

5 

Department  of  Defense  Driven 

4 

United  States  Air  Force  Driven 

3 

MAJCOM  Driven 

2 

Unit  Driven 

1 

Not  Mission  Driven 

mission  priority  for  integration  efforts  at  the  interface.  The  scale  for  the  number  of 
integrations  is  duplicated  from  that  used  in  the  value  function  for  minimizing  acquisition 
cost.  Linear  conditional  value  functions  scaled  between  0  and  1  were  assumed  to  the 
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simplify  the  elicitation.  Figure  18  shows  the  points  throughout  the  two  attribute  value 
space  that  were  used  to  determine  the  scaling  constants,  sw,  sn,  and  s1011 .  The  maximum 

value  of  one  is  achieved  when  there  are  20  integrations  and  a  level  six  schedule  urgency. 
The  iterative  process  detailed  in  Figure  17,  was  employed  to  determine  the  scaling 
constants  used  in  the  multilinear  value  function  for  schedule  urgency  shown  in  equation 
(17). 


Schedule  Urgency-  v10(x10) 


■  0.80-1.00 
■  0.60-0.80 

■  0.40-0.60 

■  0.20-0.40 

■  0.00-0.20 


#of  Integrations 

Vll(Xll) 


Figure  18:  Meet  Schedule  Expectations  Iso-Preference  Curves 
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(17) 


V101l(*H»*ll)  510  '  V10 (-^10 ^  +  511  ‘  V11  (-^ll)  +  51011  ’  V10 (-^10 )  ’  Vll(-*il) 

Where 

x  —  1 

Schedule  Urgency:  v10(x10)  =  — ^ — 

And 


#  of  Integrations:  vu(xn)  = ^ 


Mon  OSo  >  -Mi ) =  0-05  •  f  — — +  0. 15  •  f  —1  +  0.8  •  ^  Xw 


5  ) 


20 , 


20 


Step  5:  Weight  Value  Hierarchy 

The  value  hierarchy  enables  the  evaluation  framework  to  be  subdivided  into  many 

quantifiable  factors;  however,  all  are  not  of  equal  importance.  In  a  weighted  additive 

value  model,  the  weight  factor  allows  for  relative  importance  to  be  considered  in  the 

composite  value  score.  Of  the  many  methods  for  assessing  weight  factors,  two  were 

considered,  rank  based  weighting  and  swing  weighting.  Rank  based  weighting  can  be 

employed  when  the  SDM  has  limited  availability  because  it  has  a  less  burdensome 

elicitation  process.  Swing  weighting  accurately  reflects  the  SDMs  preference  structure 

but  requires  a  more  detailed  and  thus  longer  data  collection  process.  Local  swing 

weighting,  anchored  on  the  most  important  factor,  was  employed  for  this  research.  This 

technique  compares  the  subordinate  values  in  a  single  branch  of  the  hierarchy.  The 

subordinate  values  are  then  ranked  from  most  to  least  important  by  assessing  the  order  in 

which  the  SDM  would  swing  them  from  the  least  preferred  to  most  preferred  level.  The 
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most  important  value  would  then  be  awarded  an  arbitrary  importance  score  (m).  The 


relative  importance  of  the  other  values,  as  a  function  of  m,  would  then  be  found  using 
indifference  assessments.  This  is  repeated  for  all  levels  of  all  branches.  The  scores  are 
then  normalized  such  that  the  weights  of  the  lowest  level  values  sum  to  one  for  use  in  the 
MAVF.  The  SDM  for  this  research  was  technically  astute  and  was  subsequently  able  to 
provide  pre-normalized  local  swing  weights. 

Fundamental  Objective  Weighting:  Maximizing  the  value  of  an  open  interface 
implementation  requires  consideration  of  the  fundamental  objectives  based  on  the 
relevant  sociopolitical  environment.  The  decision  maker  was  asked  to  determine  the 
weights  for  the  fundamental  objectives  conditioned  on  the  sociopolitical  environment  of 
the  MQ-1  and  MQ-9  over  the  past  ten  years.  The  SDM’s  subjective  assessment  indicated 
that  meeting  schedule  expectations  was  preferred  to  obtaining  the  lowest  cost  or  meeting 
acquisition  performance  expectations. 


Table  15:  Fundamental  Objective  Weighting 


Value 

Local  Weight 

Minimize  Acquisition  Cost 

0.15 

Meet  Acquisition  Performance  Expectations 

0.25 

Meet  Schedule  Expectations 

0.60 

Meet  Acquisition  Performance  Expectations  Weighting:  The  evolutionary  nature 

of  technology  and  the  need  to  maintain  a  tactical  advantage  over  the  adversary  drive 

system  changes.  Accordingly,  the  SDM  placed  the  most  weight  on  Adjust  to  Change 

because  without  change  an  open  interface  has  very  little  value.  Additionally,  developing 

a  technological  advantage  is  of  little  value  if  it  cannot  be  utilize  by  the  intended  users. 

Therefore  the  second  highest  weight  was  applied  to  Support  Users.  Finally,  the 
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willingness  to  share  information,  inherent  to  an  open  interface,  conflicts  with  methods  of 
maintaining  a  technological  advantage  by  protecting  information.  The  SDM  believed 
while  the  information  protection  level  inhibited  openness  it  did  not  preclude  it  and 
subsequently  placed  the  lowest  weight  on  Protect  Information. 


Table  16:  Meet  Acquisition  Performance  Expectations  Weighting 


Value 

Local  Weight 

Adjust  to  Change 

0.45 

Protect  Information 

0.2 

Support  Users 

0.35 

Adjust  to  Change  Weighting:  The  goal  of  implementing  an  open  interface  is  to 
tolerate  change,  not  for  the  interface  itself  to  induce  change.  Therefore,  the  SDM 
assigned  the  highest  weight  to  Minimize  Interface  Change.  Adjust  to  Technology 
Change  and  Adjust  to  Threat  Change  were  both  weighted  significantly  lower.  Adjust  to 
Technology  Changes  was  weighted  slightly  lower  because  technology  changes  can  drive 
compatibility  issues  with  an  interface.  Thus  the  SDM  determined  that  a  high  level  of 
technology  change  does  not  add  as  much  value  to  an  open  interface  implementation  as  a 
high  level  threat  change. 


Table  17:  Adjust  to  Change  Weighting 


Value 

Local  Weight 

Adjust  to  Technology  Change 

0.15 

Adjust  to  Threat  Change 

0.25 

Minimize  Interface  Change 

0.6 

Support  Users  Weighting:  The  MQ-1  and  MQ-9  are  multirole  aircraft  which 

support  both  Intelligence,  Surveillance,  and  Reconnaissance  (ISR)  missions  and  Air  to 

Ground  attach  missions.  The  SDM  assessed  the  highest  weight  on  Support  Variety  of 
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Systems.  The  next  most  highly  weighted  value  was  Support  User  Community.  Finally, 
the  SDM  assigned  the  lowest  weight  to  Support  Quantity  of  Systems  because  an  open 
interface  can  still  be  valuable  if  there  are  not  multiple  functionally  equivalent  systems 
that  connect. 


Table  18:  Support  Users  Weighting 


Value 

Local  Weight 

Support  User  Community 

0.3 

Support  Quantity  of  Systems 

0.1 

Support  Variety  of  Systems 

0.6 

Global  Weights 

After  the  local  weights  were  determined  the  weights  for  each  of  the  lowest  level 
values  were  calculated.  The  value  hierarchy  including  evaluation  measures  and  global 
weights  is  shown  in  Figure  19. 


Figure  19:  Interface  Evaluation  Framework  Value  Hierarchy  Including  Global 

Weights 
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Local  Ranks 


During  the  weighting  process  the  SDM  was  asked  to  locally  rank  the  hierarchy. 
These  local  ranks  were  used  with  the  rank  sum,  rank  exponent,  rank  reciprocal  and  ROC 
weight  determination  methods.  The  value  hierarchy  including  local  ranks  is  shown  in 
Figure  20. 


Adjust  to 

i 

Adjust  to 

Minimize 

Support  User 

Support 

Support 

Technology 

Threat 

Interface 

Community 

Quantity  of 

Variety  of 

Change 

Change 

Change 

Systems 

Systems 

3 

2 

1 

2 

3 

1 

Figure  20:  Interface  Evaluation  Framework  Value  Hierarchy  Local  Ranks 


Multiattribute  Value  Function 


The  structure  of  the  value  hierarchy  and  the  lack  of  preferential  independence 


within  the  cost  and  schedule  branches  dictated  a  MAVF  of  the  form  depicted  in  equation 


(18).  Equation  (18)  is  an  additive  value  function  with  multilinear  elements  that  capture 
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(18) 

V(Xi)  =  wl-v1(xl)  +  w2  ■v1(x2)  +  wi-vi{x^)  +  wA  ■  v4  (x4 )  +  w5  -v5(x5)  + 

W6  ■v6(x6)  +  w1-v1(x1)  +  w89  •  v89 (x8 , x9 )  +  w101 !  •  v1011  (x10 ,xu) 

Where 

Xi  =(xvx2,x3,...xu) 

attribute  dependence  where  necessary.  Though  this  function  is  more  complicated  than 
the  additive  value  function,  it  provides  a  more  accurate  representation  of  the  SDM’s 
preferences.  After  all  SAVFs,  multilinear  value  functions,  and  weight  factors  were 
determined  the  final  MAVF,  shown  in  equation  (19),  was  constructed  to  determine  the 
value  of  each  interface  implementation. 

(19) 

V(Xi)  =  0.0169-  Vj(Xj)  +  0.0281  •  v2(x2)  +0.0675-  v3(x3)  +  0.05-  v4(x4)  +  0.0263  •  v5(x5)  + 

0.0088- v6(x6)  +0.0525  -v7(x7)  +  0.15  -(0.0-  v8(x8)  +  0.05-  v9(x9)  +  0.95-  v8(x8)-v9(x9))  + 
0.60  •  (0.05 -v10(x10)  +  0. 15  •v11(x11)  + 0.80 -v10(x10)-v11(x11)) 

Where 

Xi  =  (x1,x2,x3,...x11) 

V(Xi)  =  Value  of  an  open  interface  implementation  for  scenario  X 
x,  =  Attribute  i  of  scenario  X 

v  .(xf)  =  Component  value  score  for  attribute  i  of  scenario  X 

Hierarchy  Quality  Evaluation 

A  subjective  quality  assessment  was  conducted  based  on  six  factors  described  in 
Step  2  of  Chapter  2:  Completeness,  Non-Redundancy,  Decomposability,  Operability, 
Conciseness  and  Input  Quality.  Each  factor  was  assessed  on  a  scale  from  1-4  as 
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described  in  Table  19.  The  spider-web  diagram,  Figure  21,  is  a  graphical  representation 
of  the  assessment.  The  rationale  used  for  assessing  each  factor  is  described  below. 


Table  19:  Hierarchy  Quality  Rating  Scale 


Rating 

Description 

1 

No  issues  identified  with  subject  factor 

2 

Minor  issues  identified  with  subject  factor 

3 

Major  issues  identified  with  subject  factor 

4 

Factor  not  considered  during  hierarchy  development 

Completeness 

1 

2  L 

Input  Quality  Non-Redundancy 

3 

4 

Conciseness  Decomposability 

Operability 


Figure  21:  Hierarchy  Quality  Evaluation 


Completeness :  The  subjective  assessment  of  completeness  resulted  in  a  score  of 
two.  Multiple  resources  were  consulted  from  academia,  doctrine  and  personal 
communication  to  develop  an  exhaustive  list  of  evaluation  factors.  The  personal 
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communication  element  of  the  investigation  focused  solely  on  a  UAS  system  program 
office.  Thus  any  factors  not  identified  in  the  academic  and  doctrinal  examination  would 
be  specific  to  medium  altitude  UAS  acquisition. 

Non -Redundancy :  The  value  hierarchy  was  scored  a  one  for  non-redundancy. 
There  were  ten  value  measures  considered  for  the  value  hierarchy  that  contributed  the 
nine  different  component  values.  The  number  of  integrations  is  used  in  both  the 
Minimize  Acquisition  Cost  and  Meet  Schedule  Expectations  values.  The  SDM  believed 
that  the  number  of  integrations  was  relevant  to  both  values  but  it  would  contribute 
differently  to  each  therefore  this  was  not  considered  an  issue.  Further  support  for  the  use 
of  common  measures  across  multiple  upper  level  objectives  can  be  found  in  the  paper  by 
Merrick,  Parnell,  Barnett,  and  Garcia  describing  their  analysis  of  the  Upham  Brook 
Watershed  (2005). 

Decomposability:  The  subjective  assessment  resulted  in  a  score  of  two  for 
decomposability.  Two  of  the  nine  lowest  level  values  in  the  final  hierarchy  employed  a 
multilinear  functional  form  to  capture  interactions  between  value  measures.  This  is 
considered  a  minor  issue  because  while  the  hierarchy  could  not  be  fully  decomposed  to 
independent  elements  the  dependent  elements  were  captured  with  multilinear  functions. 

Operability:  The  value  hierarchy  was  scored  two  for  operability.  The  intended 

users  of  the  interface  evaluation  framework  are  decision  makers  within  the  acquisition 

community.  All  values  and  value  measures  were  selected  based  on  direct  input  from  an 

IPT  and  SDM  from  the  acquisition  community.  During  the  scoring  process  it  was 

identified  that  many  of  the  cost  differential  estimates  were  outside  the  bounds  of  the  scale 

for  this  value.  The  result  of  this  issue  is  that  the  framework  does  not  show  great 
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sensitivity  to  cost.  Any  scenario  with  cost  to  implement  an  open  interface  that  was  more 
than  double  the  cost  to  implement  a  closed  interface  received  the  same  score.  Further 
research  would  be  necessary  to  determine  if  the  cost  scale  needs  to  be  adjusted.  If  an 
adjustment  were  necessary,  the  swing  weights  would  also  need  to  be  revisited.  This  issue 
was  considered  minor  for  operability  because  the  scale  was  well  understood  by  the 
scoring  official  but  needs  to  be  refined.  All  other  measures  demonstrated  good 
operability. 

Conciseness :  The  conciseness  of  the  hierarchy  was  assessed  a  score  of  one.  The 
lowest  level  values  did  not  indicate  any  conceptual  overlap. 

Input  Quality.  The  hierarchy  development  leveraged  silver  and  gold  inputs  to 
develop  an  initial  draft.  Platinum  inputs  were  leveraged  to  aggregate  and  refine  the  draft 
hierarchy  to  arrive  at  the  final  product.  The  extensive  use  of  SDM  inputs  provides  for 
strong  input  quality;  however  the  breadth  of  input  was  limited  to  a  single  platform  type 
and  mission  area.  The  input  quality  was  scored  a  two  because  of  the  limited  breadth  of 
platinum  standard  inputs. 

Summary 

Chapter  3  provided  a  detailed  overview  of  the  methodology  that  was  utilized  to 
collect  and  analyze  data  in  support  of  the  research  objectives.  The  first  step  in  the  VFT 
process,  problem  definition,  was  discussed.  The  chapter  then  outlined  the  data  required, 
method  of  collection  and  method  of  analysis  for  the  following  VFT  process  steps:  Create 
Value  Hierarchy  (Step  2),  Develop  Evaluation  Measures  (Step  3),  Create  Value 
Functions  (Step  4),  Weight  Value  Hierarchy  (Step  5).  Alternative  Generation, 
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Alternative  Scoring,  Deterministic  Analysis  and  Sensitivity  Analysis  (Steps  6-9)  will  be 
discussed  in  Chapter  4.  The  final  step,  Conclusions  and  Recommendations  (Step  10)  is 
addressed  in  Chapter  5. 
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IV.  Analysis  and  Results 


Chapter  Overview 

This  chapter  discusses  the  results  obtained  from  the  application  of  the  interface 
evaluation  framework  to  historical  interface  decision  scenarios,  and  the  associated 
sensitivity  analysis.  Alternatives  were  selected  to  capture  a  cross  section  of  interface 
decisions  made  on  the  MQ-9  program.  Subject  matter  expert  inputs  were  used  to  obtain 
value  scores  on  fifteen  alternatives,  interface  scenarios,  from  the  early  stages  of  the  MQ- 
l/MQ-9  UAS  programs.  The  chapter  begins  with  a  discussion  of  the  alternatives.  This  is 
followed  by  a  description  of  the  scoring  procedure  and  an  examination  of  the  resulting 
scores,  relevant  assumptions,  and  observations.  Next,  the  implications  of  the  value  scores 
are  explained.  Finally,  the  results  of  the  sensitivity  analysis,  areas  of  sensitivity,  and  a 
comparison  of  decision  factors  are  described. 

Step  6:  Alternative  Identification 

The  evaluation  framework  was  developed  with  the  goal  of  identifying  interfaces 
that  would  benefit  from  the  from  open  interface  implementation.  Therefore  an  interface 
scenario  that  receives  a  high  score  would  be  a  good  candidate  for  the  use  of  open  IIMs. 
Conversely,  an  interface  scenario  that  receives  a  low  score  would  be  a  good  candidate  for 
the  use  of  closed  IIMs.  This  research  divides  the  choices  available  to  the  SDM  for  any 
interface  scenario  into  four  categories  of  action  taken  which  correlate  with  the  four  model 
recommendation  categories:  implemented  open ,  implemented  closed,  invested  in  open, 
and  considered  open.  The  goal  of  interface  scenario  selection  was  to  capture  a  cross 
section  of  different  categories  using  a  subset  of  the  interfaces  that  exist  on  the  MQ-9 
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platform.  The  resulting  sample  data  included  a  set  of  fifteen  interface  implementation 


scenarios.  The  implemented  open  category,  where  the  a  program  office  chose  to 
implement  the  most  mature  standard  that  existed,  proved  to  be  uncommon  which  resulted 
in  only  two  identified  scenarios.  It  was  suspected  that  this  is  related  to  the  cutting  edge 
nature  of  military  systems.  By  the  time  an  interface  has  reached  full  maturity  there  are 
less  mature  higher  performance  interfaces  available.  The  implemented  closed  category, 
where  the  program  office  chose  to  implement  an  IML  1  or  IML  2  interface  standard,  was 
not  intuitive  and  resulted  in  only  two  identified  scenarios.  The  invested  in  open  category, 
where  the  program  office  chose  to  invest  resources  to  document  or  mature  a  closed 
interface,  were  very  common  which  resulted  in  ten  identified  scenarios.  Finally,  the 
considered  open  category,  where  a  mature  standard  existed  yet  the  program  office  chose 
to  implement  a  less  mature  interface,  were  infrequent  and  resulted  in  only  one  scenario. 
Due  to  the  nature  of  the  military  capabilities  involved  with  these  fifteen  interfaces,  all 
scenarios  will  be  referred  to  by  an  interface  number  and  a  designation  as  either  electrical 
(E)  or  mechanical  (M).  A  description  of  the  15  interface  scenarios  is  provided  in  Table 
20. 


Table  20:  Interface  Scenario  Descriptions 


Interface 

Scenario 

Description 

1M 

Communications  Mechanical  Interface 

IE 

Communications  Electrical  Interface 

2M 

Mission  System  Mechanical  Interface 

2E 

Mission  System  Electrical  Interface 

3M 

Mission  System  Mechanical  Interface 

3E 

Mission  System  Electrical  Interface 

4M 

Safety  System  Mechanical  Interface 

4E 

Safety  System  Avionics  Electrical  Interface 
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5M 

Data  Transmission  System  Mechanical  Interface 

5E 

Data  Transmission  System  Electrical  Interface 

6M 

Mission  System  Mechanical  Interface 

6E 

Mission  System  Electrical  Interface 

7E 

Ground  Station  Peripheral  Electrical  Interface 

8E 

Mission  System  Electrical  Interface 

9E 

Avionics  Electrical  Interface 

Step  7:  Alternative  Scoring 
Scoring  Procedure 

The  scoring  procedure  involved  examination  of  each  historical  interface  scenario 
against  the  value  measures  captured  in  the  model.  A  high  level  systems  engineer  from 
the  Medium  Altitude  UAS  System  Program  Office  was  chosen  to  perform  the  assessment 
because  he  possessed  both  access  to  the  necessary  information  and  experience,  within  the 
program  office,  with  all  aspects  of  model.  The  scoring  official  was  provided  an  Excel- 
based  evaluation  tool,  which  provided  scales,  descriptions,  and  sliding  scales  for  each 
measure.  The  inputs  to  the  model  captured  the  actual  occurrences  covering 
approximately  a  ten  year  period  leading  up  to  the  research  period.  The  scoring  official 
was  asked  to  provide  his  best  assessment  for  each  measure.  Estimates  were  utilized 
where  necessary  to  work  within  the  time  constraints  of  the  research. 

Step  8:  Deterministic  Analysis 

The  weighted  value  scores  obtained  from  the  historical  interface  scenario 
assessment  conducted  by  the  scoring  official  are  shown  in  Table  21.  The  maximum 
value  interface  scenario  shown  on  the  top  line  of  the  left  column  represents  an  interface 
scenario  in  which  the  maximum  score  was  achieved  in  all  value  measures.  Additionally, 
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Table  21:  Weighted  Value  Scores 


Interface 

Scenario 

Weighted  Value 
Score  (V(Xt)) 

Maximum  Value 

1.000 

8  E 

0.4049 

6  M 

0.2820 

6  E 

0.2535 

7  E 

0.2277 

9  E 

0.1320 

5  M 

0.1199 

5  E 

0.1199 

Interface 

Scenario 

Weighted  Value 
Score  ( V(Xi )) 

4  M 

0.0984 

4  E 

0.0984 

1  M 

0.0855 

1  E 

0.0855 

2  M 

0.0843 

2  E 

0.0843 

3  M 

0.0809 

3  E 

0.0809 

Figure  22  provides  a  graphical  depiction  of  the  contribution  of  each  of  the  component 
values  to  the  weighted  value  score  for  each  of  the  historical  scenarios.  Each  interface 
scenarios  is  represented  by  an  interface  number  followed  by  an  E,  for  electrical 
component,  or  M,  for  mechanical  component  of  the  interface.  While  none  of  the 
interface  scenarios  scored  particularly  high,  the  value  scores  indicate  that  the  electrical 
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■  Meet  Schedule  Expectations 

■  Minimize  Interface  Change 

■  Support  Quantity  of  Systems 


■  Adjust  to  Technology  Change 

■  Protect  Information 

■  Support  Variety  of  Systems 


■  Adjust  to  Threat  Change 

■  Support  User  Community 

■  Minimize  Acquisition  Cost 


Figure  22:  Open  Interface  Implementation  Value  Breakout 


component  of  interface  scenario  eight  (8E)  held  the  most  value  for  an  open  interface 
implementation.  Relevant  assumptions  and  observations  for  each  of  the  value  measures 
are  described  in  the  following  sub-sections. 

Scoring  Assumptions  and  Observations 

The  IEF  was  developed  to  aid  SDMs  with  interface  decisions  based  on  a  ten  year 
planning  horizon.  This  means  that  if  the  IEF  were  used  as  intended,  the  scoring  official 
would  be  providing  scores  based  on  what  he/she  believed  would  occur  over  the  ten  years 
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following  the  decision.  The  assumptions  and  observations  captured  below  are  based  on  a 
historical  data  set.  Subsequently,  the  data  captures  what  occurred  over  the  past  ten  years. 

Number  of  Functionally  Different  Connected  Systems:  For  this  value  measure  the 
scoring  official  counted  the  number  of  functionally  different  systems  that  were  connected 
to  the  interface  over  the  past  ten  years.  To  assess  this  and  several  other  scores  the  scoring 
official  must  delineate  between  the  host,  the  system  that  is  being  connected  to,  and  the 
tenant(s),  the  system(s)  that  are  connected  to  the  host.  The  term  connected  systems  is 
referring  to  the  tenant  system(s)  as  determined  by  the  scoring  official.  Two  connected 
systems  were  considered  functionally  different  if  they  connected  at  the  interface,  yet  the 
requirements  for  the  systems  were  different.  For  example,  if  the  SDM  expected  to 
connect  a  printer  and  a  scanner  to  the  same  interface,  a  score  of  two  would  be  obtained. 
This  is  because  the  technical  requirements  met  by  the  printer  would  clearly  be  different 
than  those  met  by  the  scanner.  Interface  scenario  8E,  6E,  and  6M  obtained  a  score  of 
twelve,  four,  four  respectively.  All  of  the  other  interface  scenarios  scored  either  a  one  or 
two  on  this  measure,  while  the  maximum  possible  score  was  a  twenty.  This  result  was 
not  unexpected.  The  attribute  scale  needs  to  capture  the  majority  of  possible  outcomes; 
however,  interfaces  that  support  many  functionally  different  systems  are  not  as  prevalent 
as  those  that  support  one  or  two. 

Average  Number  of  Functionally  Equivalent  Connected  Systems:  The  scoring 

official  calculated  the  average  number  of  functionally  equivalent  systems  that  were 

connected  to  the  interface  over  the  past  ten  years.  Two  connected  systems  were 

considered  functionally  equivalent  if  both  systems  met  or  exceeded  the  same  set  of 

performance  requirements.  Knowledge  of  the  number  of  functionally  different  connected 
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systems  was  required  prior  to  scoring  this  measure.  From  the  example  above,  if  five 
different  printers  were  available  to  meet  the  printer  technical  requirements  and  three 
different  scanners  were  available  to  meet  the  scanner  technical  requirements  then  a  score 
of  four  would  be  obtained. 

It  was  expected  and  confirmed  through  scoring  that  all  of  the  interface  scenarios 
scored  low  in  this  measure.  Systems  exist  that  have  many  comparable,  functionally 
equivalent,  replacements.  However,  it  is  suspected  that  these  systems  do  not  often 
dovetail  with  the  very  specific,  high  performance  requirements  of  a  long  endurance, 
medium  altitude  UAS. 

Average  TRL  of  Connected  Systems:  This  value  measure  required  the  scoring 
official  to  subjectively  assess  the  Technology  Readiness  Level  of  the  systems  connected 
at  the  interface.  The  TRL  for  each  of  the  connected  systems  were  then  averaged  to  obtain 
a  score. 

Threat  Environment'.  For  this  value  measure  the  scoring  official  made  a 
subjective  determination  of  the  threat  environment  that  would  be  impacting  development 
over  the  past  ten  years.  Because  all  historical  interface  scenarios  were  coming  from  the 
same  time  period  it  was  expected  that  a  common  score  would  be  obtained  for  all  fifteen 
interface  scenarios. 

Interface  Maturity.  The  scoring  official  examined  the  interfaces  standards 

available  for  implementation  at  the  time  of  decision.  The  maximum  maturity  level  of  the 

available  interface  standards  dictated  the  value  score.  Twelve  of  the  fifteen  interface 

scenarios  indicated  that  the  maturity  of  the  interface  standards  available  at  the  time  of  the 

decision  were  level  1,  where  no  defined  standard  existed  and  changes  were  controlled  by 
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the  integrator.  The  interface  standards  available  for  the  remaining  three  scenarios  were 
documented  and  controlled  by  the  Department  of  Defense  or  a  commercial  standards 
agency  and  thus  were  assessed  as  a  level  4  IML. 

Information  Protection  Level :  For  this  value  measure  the  scoring  official  assessed 
the  highest  IPL  of  the  connected  systems.  All  but  one  of  the  scenarios  under 
consideration  were  assessed  as  having  a  maximum  IPL  of  either  Secret  or  Unclassified 
FOUO.  The  remaining  scenario,  8E,  had  a  maximum  IPL  of  Compartmentalized  Top 
Secret. 

User  Community  of  Connected  Systems:  This  value  measure  required  the  scoring 
official  to  examine  the  user  community  of  the  connected  systems.  The  value  score 
captured  the  user  community  for  all  connected  systems.  If  all  of  the  users  of  the 
connected  systems  came  from  the  same  unit  then  a  value  score  of  one  would  be  awarded. 
If  the  users  came  from  different  units  but  all  within  the  USAF  then  a  value  score  of  three 
would  be  awarded.  All  of  the  historical  scenarios  under  consideration  came  from  the 
MQ-9  UAS  Air  Vehicle.  The  MQ-9  is  a  Foreign  Military  Sales  (FMS)  asset  that  is 
shared  with  other  countries.  Because  of  the  FMS  status  of  the  Air  Vehicle  all  scenarios 
were  scored  a  level  5  for  this  value  measure. 

Number  of  Integrations  at  the  Interface:  For  assessing  historical  data,  the  scoring 
official  looked  at  the  number  of  integrations  that  were  performed  at  the  interface  over  the 
past  ten  years.  If  using  the  evaluation  framework  to  examine  a  current  decision  the 
scoring  official  would  assess  the  number  integrations  based  on  planned  upgrades, 
evolutions,  and/or  system  integrations. 
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Cost  Differential :  This  value  measure  required  the  scoring  official  to  determine 


the  cost  to  implement  a  closed  interface  and  the  cost  to  implement  an  interface  with  the 
highest  interface  maturity  available.  The  two  costs  were  used  to  calculate  the  ratio  of 
cost  difference  to  the  cost  to  implement  a  closed  interface.  This  ratio  provided  the  value 
score.  If  the  highest  IML  available  was  a  level  1  or  level  2  then  the  cost  to  obtain  a 
government  owned  ICD  would  be  used.  In  all  of  the  historical  scenarios  detailed  cost 
data  was  unavailable.  A  subject  matter  expert  estimate  was  used  in  lieu  of  detailed  cost 
data.  For  future  evaluations  it  is  likely  that  detailed  cost  information  would  be  available 
for  assessment  of  this  value  score.  For  twelve  of  the  fifteen  scenarios  the  scoring  official 
determined  that  the  cost  to  obtain  a  government  owned  ICD  was  more  than  double  the 
cost  of  implementing  a  closed  interface.  In  scenario  6E  the  scoring  official  indicated  that 
the  cost  of  additional  hardware  required  to  support  implementing  an  IML  4  interface  was 
more  than  double  the  cost  of  implementing  a  closed  interface.  The  scale  for  cost 
differential  did  not  account  for  costs  of  this  magnitude  and  thus  the  maximum  score  of 
one  was  awarded  for  all  scenarios  indicating  a  component  value  score  of  zero.  The 
remaining  two  scenarios  in  which  an  IML  4  was  available,  the  scoring  official  indicated 
that  the  cost  differential  was  zero  indicating  a  component  value  score  of  one. 

Schedule  Urgency:  This  value  measure  required  the  scoring  official  to  examine 

the  schedule  urgency  of  integrations  over  the  past  ten  years  based  in  mission  priority. 

Because  each  of  the  integrations  could  have  a  different  mission  priority,  the  scoring 

official  was  asked  to  provide  an  overall  assessment  of  the  mission  priority  of  integrations 

at  the  interface.  Though  there  are  many  users  of  the  MQ-1  and  MQ-9  UASs,  many 

system  changes,  whether  driven  by  technology  change  or  threat  change,  are  filtered 
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through  a  single  unit  which  prioritizes  modifications  and  provides  direction  for  the 
program  office.  This  prioritization  resulted  in  eleven  of  the  fifteen  scenarios  being  scored 
a  Level  2  for  schedule  urgency. 

Implications  of  the  Value  Score 

The  IEF  allows  the  SDM  to  systematically  obtain  a  value  score  for  an  interface. 
The  question  remains,  “How  does  one  use  the  value  score  information  to  make  a  decision 
about  interface  implementation?”  Figure  23  provides  a  means  of  interpreting  the  value 
scores  by  comparing  them  to  the  IMF  of  available  interface  standards.  The  threshold 
i  t - - 
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Figure  23:  Value  Score  Interpretation 
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value  score  is  set  to  0.268  based  on  historical  decision  data.  The  Implement  Closed 
quadrant  is  associated  with  scenarios  that  obtain  little  value  for  the  implementation  of  an 
open  interface  and  available  interface  standards  are  at  an  IML  1  or  2.  In  other  words, 
there  is  not  a  business  case  for  an  open  interface  and  the  standards  are  immature.  The 
Implement  Open  quadrant  is  associated  with  scenarios  that  obtain  high  value  for  an  open 
interface  and  interface  standards  that  are  at  an  IML  3  or  4.  This  indicates  that  there  is  a 
strong  business  case  for  an  open  interface  and  documented,  controlled  interface  standards 
are  available.  The  Invest  In  Open  quadrant  is  associated  with  scenarios  that  obtain  a  high 
value  for  the  implementation  of  an  open  interface  but  documented,  controlled  interfaces 
are  not  available.  This  situation  would  suggest  to  the  SDM  that  investment  in  developing 
or  maturing  the  interface  standard  may  be  a  worthwhile  endeavor.  Finally,  the  Consider 
Open  quadrant  is  associated  with  scenarios  that  obtain  low  value  for  an  open  interface  but 
IML  3  or  4  standards  are  available.  This  situation  would  suggest  that  use  of  an  open 
interface  is  preferred  if  no  additional  resources,  time  or  money,  are  required.  This 
graphic  is  only  meant  to  provide  guidance  to  the  decision  maker.  The  value  scores  and 
IML  levels  of  the  historical  interface  scenarios  are  indicated  by  red  circles  in  Figure  23. 
Table  22  provides  an  examination  of  the  model  recommendation,  actual  implementation 
decision,  and  subjective  commentary  on  discrepancies  for  each  of  the  historical  scenarios. 
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Table  22:  Model  Recommendation  Vs.  Actual  Action  Taken 


Interface 

Scenario 

Model 

Recommendation* 

Action 

Taken 

Comments/Rationale 

1  M 

Implement  Closed 

Invested  in  Open 
at  IML  3 

The  Program  Office  invested  in  ICDs  based  on  major 
system  interfaces.  Guiding  documents  do  not  provide 
explicit  methods  or  metrics  for  business  case  analysis 
of  key  interfaces  (IPT  Engineering,  personal 
communication,  lanuary  16,  2014). 

1  E 

Implement  Closed 

Invested  in  Open 
at  IML  3 

2  M 

Implement  Closed 

Invested  in  Open 
at  IML  3 

2  E 

Implement  Closed 

Invested  in  Open 
at  IML  3 

3  M 

Implement  Closed 

Invested  in  Open 
at  IML  3 

3  E 

Implement  Closed 

Invested  in  Open 
at  IML  3 

4  M 

Implement  Closed 

Invested  in  Open 
at  IML  3 

4  E 

Implement  Closed 

Invested  in  Open 
at  IML  3 

5  M 

Implement  Closed 

Invested  in  Open 
at  IML  3 

5  E 

Implement  Closed 

Invested  in  Open 
at  IML  3 

6  M 

Implement  Open 

Implemented  Open 
at  IML  4 

No  Discrepancy 

6  E 

Consider  Open 

Considered  Open  at 
IML  4  but 
implemented  IML  3 

An  IML  4  MIL-STD  existed  however  the  cost  to 
implement  fully  was  prohibitive.  The  choice  was 
made  to  implement  a  tailored  version  of  the  MIL- 
STD.  The  implementation  of  an  IML  3  interface  is  in 
line  with  the  model  recommendation  (IPT 

Engineering,  personal  communication,  lanuary  16, 
2014). 

7  E 

Consider  Open 

Implemented  Open 
at  IML  4 

An  IML  4  commercial  standard  existed  and  was 
implemented  at  no  added  cost  (IPT  Engineering, 
personal  communication,  lanuary  16,  2014).  The 
implementation  of  an  IML  4  interface  is  in  line  with 
the  model  recommendation. 

8  E 

Invest  In  Open 

Implemented  Closed 
at  IML  1 

The  Program  Office  is  currently  investigating  the 
implementation  of  an  IML  3  interface.  Information 
available  at  the  time  of  program  planning  did  not 
indicate  a  need  for  an  open  interface.  However, 
numerous  integrations  were  added  to  the  program  plan 
to  meet  various  mission  needs  (IPT  Engineering, 
personal  communication,  lanuary  16,  2014). 

9  E 

Implement  Closed 

Implemented  Closed 
at  IML  1 

No  Discrepancy 

*Model  recommendation  is  based  on  a  Threshold  Value  Score  of  0.268 
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Step  9:  Sensitivity  Analysis 

The  goal  of  sensitivity  analysis  is  to  examine  the  impact  of  input  changes  on  the 
output  and  recommendations  of  the  model.  The  various  sensitivity  analyses  conducted 
on  the  IEF  are  described  below.  First,  a  weighting  technique  comparison  was  conducted 
to  examine  the  effect  of  different  weighting  techniques  on  the  model  output.  Following 
the  weighting  technique  comparison  a  rank  order  sensitivity  analysis  explored  the  impact 
of  weight  variations  on  the  rank  order  of  alternatives.  Next,  a  value  threshold  sensitivity 
analysis  captured  the  areas  of  sensitivity  to  an  established  threshold  associated  with  a 
open/closed  implementation  decision  point.  Finally,  an  exploration  of  the  impact  of 
changing  the  bounds  on  the  number  of  integrations  value  measure  on  the  decision 
threshold  was  conducted. 

Weight  comparison 

Swing  weighting,  an  indirect  weighting  technique,  was  used  to  establish  the 
weights  for  the  evaluation  framework  multiattribute  value  function  (MAVF).  However, 
several  other  direct  weighting  techniques  using  swing  ranks  were  considered.  Table  23 
shows  a  comparison  of  the  swing  weights  to  the  weights  that  would  have  been  obtained  if 
the  local  swing  ranks  were  utilized  with  each  of  the  direct  techniques  to  calculate  global 
weights.  The  table  shows  great  consistency  between  the  various  techniques.  Though  the 
table  shows  consistency,  in  many  cases  it  only  takes  small  variations  in  the  weights  to 
effect  the  rank  order  of  alternatives.  Table  24  shows  the  rank  order  that  would  have  been 
indicated  with  each  of  the  different  weighting  techniques.  The  rank  order  remains 
generally  consistent  across  all  of  the  weighting  techniques.  The  only  area  of 
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inconsistency  is  highlighted  in  grey.  The  Rank  Sum  indicated  an  order  change  between 


scenario  6E  and  scenario  7E. 


Table  23:  Weight  Variations  by  Technique 


Weighting  Technique 

Value 

Swing  Weight 

Rank  Sum 

Rank  Exponent 

Rank  Reciprocal 

Rank  Order 

Centroid 

Meet  Schedule  Expectations 

0.600 

0.500 

0.600 

0.545 

0.611 

Adjust  to  Technology  Change 

0.017 

0.028 

0.017 

0.027 

0.019 

Adjust  to  Threat  Change 

0.028 

0.056 

0.056 

0.041 

0.047 

Minimize  Interface  Change 

0.068 

0.083 

0.110 

0.081 

0.104 

Protect  Information 

0.050 

0.056 

0.029 

0.050 

0.031 

Support  User  Community 

0.026 

0.037 

0.028 

0.020 

0.021 

Support  Quantity  of  Systems 

0.009 

0.019 

0.009 

0.014 

0.009 

Support  Variety  of  Systems 

0.053 

0.056 

0.056 

0.041 

0.047 

Minimize  Acquisition  Cost 

0.150 

0.167 

0.096 

0.182 

0.111 

Table  24:  Alternative  Ranks  According  to  Weighting  Technique 


Weighting  Technique 

Rank 

Swing  Weight 

Rank  Sum 

Rank  Exponent 

Rank  Reciprocal 

Rank  Order 

Centroid 

1 

8  E 

8  E 

8  E 

8  E 

8  E 

2 

6  M 

6  M 

6  M 

6  M 

6  M 

3 

6  E 

7  E 

6  E 

6  E 

6  E 

4 

7  E 

6  E 

7  E 

7  E 

7  E 

5 

9  E 

9  E 

9  E 

9  E 

9  E 

6 

5  M 

5  M 

5  M 

5  M 

5  M 

7 

5  E 

5  E 

5  E 

5  E 

5  E 

8 

4  M 

4  M 

4  M 

4  M 

4  M 

9 

4  E 

4  E 

4  E 

4  E 

4  E 

10 

1  M 

1  M 

1  M 

1  M 

1  M 

11 

1  E 

1  E 

1  E 

1  E 

1  E 

12 

2  M 

2  M 

2  M 

2  M 

2  M 

13 

2  E 

2  E 

2  E 

2  E 

2  E 

14 

3  M 

3  M 

3  M 

3  M 

3  M 

15 

3  E 

3  E 

3  E 

3  E 

3  E 

Rank  Sensitivity 

The  robustness  of  the  model  was  examined  through  a  single  factor  sensitivity 
analysis.  The  goal  of  this  analysis  was  to  explore  the  impact  of  changes  in  weight  factors 


93 


on  the  rank  order  and  overall  value  score  of  alternatives.  A  summary  of  the  results  of  the 
analysis  is  provided  in  this  section.  The  full  results,  in  graphical  form,  can  be  found  in 
Appendix  B:  Sensitivity  Analysis.  Each  graph  is  an  examination  of  a  single  weight 
factor.  The  red  vertical  line  indicates  the  weight  determined  by  swing  weighting.  The  x- 
axis  represents  the  weight  factor,  varying  from  zero  to  one,  and  the  y-axis  indicates  the 
weighted  value  score,  varying  from  zero  to  one.  Each  of  the  fifteen  historical  interfaces 
scenarios  are  shown  in  a  different  color.  The  intersection  of  the  red  vertical  line  and 
interface  scenario  line  illustrates  the  weighted  value  score  obtained  under  the  assessed 
swing  weights  described  in  Chapter  3.  Moving  to  the  left  (right)  of  the  red  line  indicates 
the  effect  of  a  decrease  (increase)  in  weight  on  the  value  score. 

It  was  expected  that,  due  to  the  variation  in  component  value  scores  of  many  of 
the  historical  interface  scenarios,  the  evaluation  framework  would  exhibit  sensitivity  to 
changes  in  weight  for  many  of  the  values.  The  values  that  showed  sensitivity  to  weight 
changes  were,  Meet  Schedule  Expectations,  Minimize  Interface  Change,  Protect 
Information,  Support  Quantity  of  Systems,  and  Minimize  Acquisition  Cost.  As  can  be 
seen  in  Figure  24  scenario  8E  holds  the  highest  value  score  at  the  elicited  weights. 
However,  if  the  weight  (0.0675)  applied  to  Minimize  Interface  Change  were  increased  to 
>0.18,  while  all  others  were  proportionally  adjusted,  the  highest  valued  alternative  would 
change  to  scenario  6M. 
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Sensitivity  Analysis  of  Changing  the 
Weight  of  Minimize  Interface  Change 
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Figure  24:  Sensitivity  Analysis  of  Changing  the  Weight  Applied  to  the  Minimize 

Interface  Change  Value 


Value  Threshold  Sensitivity 

The  scenario  value  score  as  it  relates  to  the  threshold  value  score  is  the  primary 
focus  of  the  evaluation  framework  because  it  provides  the  SDM  direction  on  which 
interfaces  should  employ  an  open  interface  and  which  should  not.  Above  this  threshold 
the  SDM  should  choose  to  employ  open  interface  standards  if  possible.  Conversely, 
below  this  threshold  the  SDM  should  choose  to  employ  a  closed  interface.  This 
sensitivity  analysis  examines  the  impact  of  adjustments  to  the  weights  that  would  cause  a 
particular  alternative  to  rise  above  or  fall  below  the  threshold  of  interest.  The  threshold 
value  score,  indicated  by  a  horizontal  red  line,  was  added  to  the  graph  described  in  the 
previous  section.  Figure  25  shows  an  example  using  a  threshold  value  score  of  0.268. 
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Figure  26  shows  a  magnified  view  of  the  black  outlined  section  of  Figure  25.  As  can  be 

seen  in  Figure  26,  at  the  current  assessed  weight  only  two  interface  scenarios  had  value 

scores  that  rose  above  the  threshold  value  score.  This  indicated  that,  of  the  scenarios 

Sensitivity  Analysis  of  Changing  the 
Weight  of  Support  User  Community 
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Figure  25:  Example  Value  Score  Sensitivity 


under  consideration,  only  8E  and  6M  should  employ  open  interfaces.  However,  if  the 
weight  applied  to  the  Support  User  Community  value  were  raised  to  0.12625  from 
0.02625,  the  model  showed  that  four  scenarios  were  above  the  representative  threshold 
value  score.  When  sensitivities  of  this  nature  are  present  the  SDM  is  advised  to  closely 
examine  the  subject  weights  before  final  decisions  are  made. 
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Figure  26:  Exploded  View  of  Figure  25 


This  research  attempted  to  identify  a  threshold  value  score  for  open  interface 
decisions  based  on  very  limited  historical  information.  Therefore  the  value  threshold 
sensitivity  analysis  was  conducted  with  a  score  of  0.268.  A  brief  overview  of  the 
findings  is  provided,  while  the  full  analysis,  in  graphical  form,  is  provided  in  Appendix 
B:  Sensitivity  Analysis.  It  was  expected  that  many  of  the  attributes  would  exhibit  value 
threshold  sensitivity  because  there  was  only  a  small  difference  between  the  highest  and 
lowest  value  score.  The  analysis  showed  that  a  <10%  change  to  the  weights  applied  to 
six  of  the  nine  attributes  would  result  in  scenario  6E  rising  above  the  value  threshold. 
The  attributes  that  were  not  included  were,  Adjust  to  Technology  Change,  Support 
Quantity  of  Systems,  and  Support  Variety  of  Systems.  Similarly,  scenario  7E  would  rise 
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above  the  value  threshold  if  a  <10%  change  occurred  to  the  weights  of  all  attributes 
except  Adjust  to  Technology  Change,  Support  Variety  of  Systems,  and  Minimize 
Acquisition  Cost.  If  weight  changes  of  this  magnitude  were  to  occur  the  model 
recommendation  for  both  6E  and  7E  would  change  from  Consider  Open  to  Implement 
Open. 

Decision  Factor  Comparison 

The  interface  decision  factors  identified  by  the  OSJTF  and  those  uncovered  by  this 
research  do  not  match  exactly,  however  many  of  the  same  concepts  are  captured. 
Conceptual  linkages  were  established  to  connect  the  interface  decision  factors  identified 
by  the  OSJTF  and  those  decision  factors  identified  through  this  research.  Figure  27 
depicts  the  OSJTF  decision  factors,  from  Table  1,  in  grey  rounded  rectangles  and  the  IEF 
decision  factors,  from  Figure  8,  in  orange  rectangles.  The  lines  connecting  the  factors 
represent  a  conceptual  overlap  between  factors  as  determined  by  a  subjective  assessment. 
The  linkages  to  OSJTF  factor  8  are  examined  as  an  example.  Figure  27  shows  that  the 
IEF  decision  factors,  Information  Protection,  User  Community,  and  Variety  of  Systems 
are  conceptually  linked  to  OSJTF  factor  8.  An  obvious  linkage  is  that  between  the 
information  protection  factor  from  the  IEF  and  the  need  for  interface  control  highlighted 
by  the  OSJTF.  A  less  obvious  link  is  between  the  user  community  factor  and  the  need  for 
interface  control.  This  linkage  captures  the  fact  that  as  the  user  community  of  an 
interface  becomes  larger  and  more  varied  the  need  for  interface  control  also,  logically, 
increases.  This  establishes  a  conceptual  link  between  the  two  factors.  Further,  there  is  a 
link  between  the  need  for  flexibility  and  modularity  and  the  variety  of  systems  that  are,  or 
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are  intended  to  be,  employed  at  an  interface.  If  there  is  a  plan  to  have  high  number  of 
functionally  different  systems  connected  to  the  same  interface,  there  is  an  apparent  need 
for  a  flexibly  interface  in  a  modular  architecture. 

Summary 

Chapter  4  provided  a  synopsis  of  the  results  that  were  obtained  and  the  analysis 
that  was  conducted  as  part  of  the  IEF  research.  Alternative  Generation  (Step  6)  and 
Alternative  Scoring  (Step  7)  were  discussed  first  followed  by  Deterministic  Analysis 
(Step  8)  and  an  explanation  of  the  implications  of  the  value  score.  The  chapter  concluded 
with  a  series  of  sensitivity  analyses  and  a  comparison  of  the  OSJTF  decision  factors  with 
those  found  during  the  IEF  research. 
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V.  Conclusions  and  Recommendations 


The  purpose  of  this  research  was  to  develop  an  evaluation  framework  and 
decision  support  tool  to  assess  the  value  of  an  open  interface  in  line  with  the  principles 
identified  in  the  OSA/MOSA  guidance.  The  initial  concentration  of  the  effort  was  to 
capture  the  deterministic  factors,  evaluation  measures  for  those  factors,  and  the  relative 
importance  of  each  factor  to  the  value  of  an  open  interface  to  construct  a  multiattribute 
value  function  (MAVF).  After  the  MAVF  was  constructed,  based  on  Silver,  Gold,  and 
Platinum  inputs,  the  hierarchy  was  reviewed  for  quality.  Finally,  historical  interface 
scenarios  were  examined  using  the  evaluation  framework. 

This  chapter  begins  with  an  explanation  of  the  significance  of  the  research. 
Following  the  significance  section,  recommendations  for  the  acquisition  community  to 
aid  in  adoption  of  the  IEF  and  recommendations  for  future  research  are  provided. 

Finally,  conclusions  found  during  the  course  of  this  research  are  described. 

Significance  of  Research 

Current  DoD  guidance  prescribes  the  use  of  the  MOSA  to  promote  OSA.  The 

OSJTF  identifies  five  principles  to  guide  the  acquisition  community  in  the  execution  of 

the  approach.  The  guidance  provides  a  broad  set  of  factors  to  consider  when  determining 

which  interfaces  warrant  the  application  of  open  standards.  However,  these  factors  lack 

defined  metrics  and  do  not  indicate  the  relative  importance  of  the  factors.  In  addition, 

there  does  not  exist  a  structured  method,  process  or  tool  to  support  interface  decisions  for 

MOSA.  This  research  attempted  to  formalize  the  OSJTF’s  broad  guidance  through  the 

development  of  a  deterministic  decision  model.  A  decision  model  of  this  nature  provides 
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a  method  for  justified  and  consistent  decision  making  in  this  area.  Further,  the  use  of  a 
decision  model  provides  leadership  the  ability  to  decentralize  interface  decision  making 
while  maintaining  the  ability  to  control  and  refine  the  process. 

Recommendations  for  Action 

Initiate  data  collection  for  IEF  refinement 

The  model  created  in  this  research  is  a  proof  of  concept  that  leverages  decision 
analysis  tools  to  structure  the  values  of  the  acquisition  community  with  respect  to  open 
interface  implementation.  The  IEF  represents  a  large  set  of  decision  factor  inputs; 
however,  the  inputs  for  evaluation  measures,  swing  weights,  and  historical  scenarios  were 
limited  to  a  single  program  office.  It  is  recommended  that  leadership  in  the  acquisition 
community  implement  a  data  collection  requirement  in  the  program  offices  based  on  the 
evaluation  measures  defined  in  this  IEF.  This  data  will  serve  many  purposes.  First,  it 
will  help  leadership  to  better  understand  the  bounds  of  the  value  measures  and  would 
allow  for  value  measure  refinement.  Additionally,  the  information  collected  will  provide 
a  means  to  refine  the  threshold  value  score  and  facilitate  adoption  of  an  evaluation  tool  of 
this  pedigree  in  the  future. 

Examine  linearity  assumption  for  interaction  terms 

The  second  recommendation  for  action  is  to  further  examine  the  linearity 
assumption  found  in  the  IEF  interaction  terms.  To  simplify  the,  already  arduous, 
elicitation  process,  linear  functions  were  assumed  for  all  multilinear  component  value 
functions.  The  author  believes  that  a  concave  or  convex  functional  form  may  be  more 
representative  of  the  SDMs  value  preferences,  but  was  not  explored  due  to  the  linearity 


102 


assumption.  The  SDM  was  able  to  solidify  his/her  relative  preferences  based  on  the 
linear  assumption;  however,  it  is  possible  that  some  rank  inconsistencies  exist. 
Regardless,  further  examination  of  this  area  would  provide  increased  confidence  in  the 
results  of  the  framework  even  if  the  linearity  assumption  remains  unchanged. 

Recommendations  for  Future  Research 
Examination  of  misaligned  recommendations 

Ten  of  the  fifteen  historical  scenarios  that  were  examined  show  a  misalignment 
between  the  model  recommendations  and  the  program  office  decision.  There  are  two 
potential  explanations  for  this  misalignment.  1)  The  original  analysis  conducted  by  the 
program  office  included  factors  not  considered  in  the  IEF.  2)  The  program  office 
decisions  were  based  on  what  they  believed  would  occur  while  the  IEF  recommendation 
is  based  on  what  actually  occurred.  The  question  remains,  “Does  acquisition  leadership 
believe  that  the  interface  implementation  decisions  that  were  made  on  the  ten  misaligned 
scenarios  were  “good”  decisions,  given  what  has  occurred?  If  the  answer  to  this  question 
is  yes,  then  additional  research  is  warranted  to  identify  the  factors,  and  associated 
structure  and  weights,  not  considered  in  the  IEF  that  would  resolve  the  misalignment.  If 
the  answer  to  this  question  is  no  then  no  additional  research  in  this  area  is  necessary. 
Application  of  the  IEF  value  hierarchy  to  other  acquisition  portfolios 

The  Air  Force  Life  Cycle  Management  Center  (AFLCMC)  is  the  organization 
responsible  for  weapon  system  acquisition  in  the  USAF.  The  organization  is  divided  into 
many  weapon  system  portfolios,  each  led  by  a  Program  Executive  Officer.  Some  of  the 
major  weapon  system  portfolios  are:  1)  Intelligence,  Surveillance  and  Reconnaissance  / 
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Special  Operations  Forces,  2)  Tankers,  3)  Fighter/Bomber,  and  4)  Mobility  (Air  Force 
Acquisition  -  Organizations).  The  value  hierarchy,  developed  as  part  of  the  IEF, 
leveraged  inputs  from  academic  research,  doctrinal  documentation,  and  subject  matter 
experts  from  the  Medium  Altitude  UAS  program  office,  part  of  the  ISR/SOF  portfolio. 
Additionally,  the  evaluation  measures  utilized  in  the  IEF  were  also  developed  purely  on 
inputs  from  the  program  office.  While  the  results  of  the  research  indicate  that  the  value 
hierarchy  corresponds  with  those  values  identified  by  the  OSJTF,  which  provides  MOSA 
guidance  for  weapon  systems,  additional  research  to  confirm  the  applicability  of  the  IEF 
to  PEO  portfolios  other  than  ISR/SOF  is  needed. 

Effect  of  acquisition  portfolio  on  swing  weights  and  value  threshold 

In  addition  to  research  into  the  applicability  of  the  hierarchy  to  other  PEO 
portfolios  it  is  also  important  to  explore  the  effect,  if  any,  a  change  in  acquisition 
portfolio  would  have  on  the  swing  weights  and  value  threshold.  The  swing  weights  and 
swing  ranks  determined  for  the  IEF  were  based  on  a  single  decision  frame.  A  frame 
which  involved  a  platform  in  the  ISR/SOF  portfolio  developed  during  a  period  of  war  in 
which  rapidly  changing  tactics  were  employed.  The  SDM  indicated  that  the  swing 
weights  applied  to  the  framework  represented  priorities  that  were  specific  to  the 
timeframe  and  the  platform  under  consideration.  Further,  the  SDM  indicated  that  if  the 
framework  were  applied  to  next  the  ten  years  rather  than  the  past  ten  years  the  weight 
applied  to  the  cost  and  performance  fundamental  objectives  would  be  significantly  higher 
than  currently  assessed.  Additional  research  into  weighting  methods  that  can 
accommodate  for  changing  portfolios  and/or  changing  acquisition  priorities  is  desirable. 
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As  described  in  the  recommendations  for  action  section  above,  analysis  of  more 
would  increase  confidence  in  the  value  threshold.  During  model  development,  two 
hypotheses  could  not  be  confirmed  with  the  available  data.  First,  the  value  threshold  may 
not  be  a  single  level,  instead  it  could  be  four  different  levels,  each  dependent  on  the  IML 
of  the  standards  available  for  implementation.  The  value  score  interpretation  shown  in 
Figure  23  could  then  be  transformed  to  that  depicted  in  Figure  28.  The  second  hypothesis 
was  that  the  value  threshold/s  would  be  common  across  all  acquisition  portfolios.  To 
confirm  this  hypothesis,  data  from  other  weapon  system  portfolios  must  be  collected  and 
analyzed. 

To  explore  the  impact  of  different  acquisition  portfolios  on  both  swing  weights 
and  value  threshold  it  is  recommended  that  a  more  expansive  data  collection  effort  is 
undertaken.  This  effort  should  include  assessment  of  historical  interface  scenarios  and 
swing  weights  from  weapon  systems  in  each  of  the  major  portfolios  covering  multiple 
threat  environments.  Additionally,  qualitative  data  should  be  collected  to  capture  any 
decision  influences,  such  as  policy,  economic,  or  political  pressures  present  at  the  time  of 
the  historical  decision. 
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Figure  28:  Alternative  Threshold  Value  Score  Interpretation 


Treatment  of  uncertainty  in  the  IEF 

The  IEF  was  developed  under  the  assumption  of  certainty.  While  certain,  or 
highly  confident,  answers  can  be  provided  for  some  of  the  value  measures  captured  in  the 
model,  many  of  the  measures  include  assessment  of  future  events  over  the  planning 
horizon.  The  assumption  of  certainty  is  not  an  issue  for  the  evaluation  of  historical 
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scenarios  where  there  is  no  uncertainty  in  the  data.  This  is  not  the  case,  when  assessing 
present  decisions.  Including  the  treatment  of  uncertainty  in  the  model  is  possible; 
however,  the  assessment  difficulty,  and  subsequent  value  to  decision  making  is  unknown. 
In  Chapter  2,  a  discussion  of  techniques  for  the  development  of  multiattribute  decision 
models,  including  uncertainty,  was  provided.  There  are  three  components  necessary  to 
support  research  in  this  area.  First,  an  examination  of  the  existing  framework  for 
uncertain  elements.  Second,  one  would  need  to  determine  if  either  of  the  techniques  for 
implementing  uncertainty  in  multiattribute  decision  models  can  be  reasonably 
implemented  while  maintaining  the  usability  of  the  tool  for  the  general  acquisition 
community.  Finally,  leveraging  the  information  from  the  first  two  components  one  could 
elicit  risk  attitude  data  from  members  of  the  acquisition  community. 

Conclusions  of  Research 

The  IIM  recommendations  made  by  the  IEF,  with  a  threshold  value  score  of 
0.268,  show  positive  correlation  to  the  decision  made  by  the  program  office  on  five  of 
fifteen  historical  interface  scenarios.  This  indicates  that  the  IEF  reflects  the  values  of 
acquisition  decision  makers.  The  remaining  ten  scenarios  indicated  a  misalignment 
between  the  model  recommendations  and  the  historic  program  office  decisions.  This 
indicates  an  area  of  further  research  to  resolve  the  discrepancy. 

Adoption  of  the  IEF  could  provide  the  acquisition  community  a  repeatable, 
justifiable  method  for  examination  of  open  interfaces.  Implementation  of  the  IEF  will 
rely  upon  additional  data  collection  to  support  threshold  value  score  refinement. 
Utilization  of  the  IEF  and  analysis  of  recommendation  accuracy  will  provide  senior 
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leaders  the  ability  to  objectively  refine  decision  weights.  Ultimately  the  IEF  will  provide 


the  acquisition  workforce  a  tool  for  OSA/MOSA  decision  making  while  simultaneously 
providing  senior  leadership  a  method  to  control,  monitor,  and  refine  the  implementation 
of  OSJTF  guidance. 


108 


Appendix  A:  Scenario  Scoring 
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Appendix  B:  Sensitivity  Graphs 
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Sensitivity  Analysisof  Changing  the 
Weight  of  Support  User  Community 
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Sensitivity  Analysis  of  Changing  the 
Weight  of  Support  Variety  of  Systems 
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Sensitivity  Analysis  of  Changing  the 
Weight  of  Meet  Schedule  Expectations 
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